Commit Graph

987 Commits

Author SHA1 Message Date
Yann E. MORIN
e9cdabee71 support/graph-size: add option to change percentage to group in Others
Currently, we group packages that contribute less then 1%, into the
"Other" category.

However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.

Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.

Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
 - remove empty base class definition from Config;
 - use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:44:27 +02:00
Peter Korsgaard
9b9abb0dd0 Update for 2019.08-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-09 09:04:28 +02:00
Brent Generous
d57e73078a Makefile: ensure $BINARIES_DIR exist before post-image scripts
When no filesystem is enabled, the $BINARIES_DIR is not created. Yet,
the post-image scripts are still run. When those want to generate an
image in there, they may fail as the dirctory does not exist (it did
exist before we started applying preparatory changes for top-level
parallel build, so scripts got to rely on that assumption).

Do in target-post-image as we do in the sdk rule: create the directory
before calling the scripts.

Signed-off-by: Brent Generous <bgenerous@impinj.com>
[yann.morin.1998@free.fr:
  - create the directory before calling the scripts
  - don't drop the creation in the sdk rule
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 17:37:56 +02:00
Yann E. MORIN
f879203cfc core: drop now-useless prepare-kconfig rule
This rule was added back in 9429e7b698 (core: introduce an intermediate
rule before the configurators) when the kconfig-side br2-external file
was generated separately from the Makefile-side one.

Now that they are generated together very early in the Makefile, we no
longer need this intermediate rule. Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
[Peter: also drop outdated reference in the manual]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
d027cd75d0 core: generate all br2-external files in one go
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.

But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.

And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.

Make it so.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
76345bf6bc core: simplify removal of generated br2-external files
Now that all the br2-external generated files are named after the same
pattern, it gets easier to remove them all using a glob.

Furthermore, we're on the verge of introducing more such generated
files, so removing them at one fell swoop will be simpler too.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
492d09bab2 core: rename generated .br-external.mk file
Now that the two (all of them!) br2-external related files are generated
in the same location, it makes sense they are named after the same
pattern.

When initial support for (then single) br2-external trees was added back
in a4239f7fd1 (core: introduce the BR2_EXTERNAL variable), it was not
clear-cut why that file was not named with a br2 prefix.

So rename it now.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
54af0551b8 core: move generated .br2-external kconfig file to $(BASE_DIR)
Currently, that file is generated rather late in the configuration
process, so BUILD_DIR is known (and exists) by then.

We're soon to generate that file much earlier, at a point where
BUILD_DIR is not yet known, so we have two options:
 1- declare BUILD_DIR earlier;
 2- generate the file in an already-known location.

We go with the second solution, as we're already generating a
br2-external related file in BASE_DIR, so we can as well generate all
br2-external files in the same place.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Atharva Lele
d590b37633 Makefile: don't export GZIP environment variable
We export GZIP = -n so that GZIP does not record original
name and timestamps. However..

GZIP environment variable is deprecated and soon will not be
supported in future GZIP versions. GZIP suggests the use of a
wrapper to pass options globally but it might be difficult to
implement in Buildroot. For now, we don't export the variable
and fix reproducibility issues per package as they show up in
Autobuilder.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-07-17 08:46:05 +02:00
Peter Korsgaard
2421c246b2 Kickoff 2019.08 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 10:17:27 +02:00
Peter Korsgaard
8d4e26da08 Update for 2019.05
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 10:01:07 +02:00
Peter Korsgaard
2c96d648a2 Update for 2019.05-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-25 23:38:05 +02:00
Peter Korsgaard
2adc578dfd Update for 2019.05-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-15 22:35:04 +02:00
Peter Korsgaard
a8b229a75c Update for 2019.05-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-08 22:54:44 +02:00
Yann E. MORIN
f5f17c4dd7 core: remove show-dependency-tree
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.

show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-07 23:06:05 +02:00
Nylon Chen
b58a6bf774 package/binutils: fix build error due to architecture name is incomplete
Fixes
  http://autobuild.buildroot.net/results/128/12803a705586e82fdfb49013da2eb3b9879ccd45/

Signed-off-by: Che-Wei Chuang <cnoize@andestech.com>
Signed-off-by: Greentime Hu <greentime@andestech.com>
Signed-off-by: Nylon Chen <nylon7@andestech.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-20 16:16:39 +02:00
Yann E. MORIN
5abb88218e core: add per-package and per-filesystem show-info
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:47:22 +02:00
Yann E. MORIN
f8177b7813 core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.

Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...

We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).

The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.

It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma:  [ { ... }, { ... } ]

So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).

Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:37:44 +02:00
Peter Korsgaard
c24faa81e8 Makefile: release: really drop build/docs from release tarball
Commit 15cb98769e (release: remove manual build files from release
tarballs) tried to remove the temporary files from the manual build from the
release tarball, but manual-clean only removes build/docs/manual and leaves
build/docs in the tarball.

Instead use 'make clean' to completely remove the build directory from the
tarball.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-29 23:09:41 +01:00
Yann E. MORIN
870f37fe04 core: add make-based full-dependency list
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.

Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.

However, make being make, already builds the whole dependency tree
information, so we can just ask for it.

Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:

    foo: target 1.2.3
    foo -> bar host-meh
    bar: target virtual
    bar -> buz
    buz: target 2.3.4
    buz ->
    host-meh: host virtual
    host-meh -> host-bleark
    host-bleark: host 3.4.5
    host-bleark ->
    rootfs-meh: host
    rootfs-meh -> host-bleark

To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.

For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):

    $ time make graph-depends
    real    0m27.344s
    $ time make show-dependency-tree
    real    0m3.848s

>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):

    $ time make graph-depends
    real    1m56.459s
    $ time make show-dependency-tree
    real    0m5.748s

There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 15:39:12 +01:00
Arnout Vandecappelle (Essensium/Mind)
fd5bd12379 Makefile: printvars: don't print anything when VARS is not set
Using 'make printvars' for printing all variables is not very useful.
E.g. all macros will output some bogus value. In addition, the same can
be achieved with 'make -p'.

We can simply remove the condition on $(VARS). If VARS is not set, the
filter expression will be empty which matches nothing, so nothing is
printed.

Note that the old behaviour can still be achieved with:
make printvars VARS=%

Update the 'make help' text to match the new behaviour.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-12 21:58:32 +01:00
Peter Korsgaard
d29ec62899 Kickoff 2019.05 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-05 10:03:32 +01:00
Peter Korsgaard
b9674056fb Update for 2019.02
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-04 22:49:56 +01:00
Peter Korsgaard
bdfea8428f Update for 2019.02-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 12:57:30 +01:00
Peter Korsgaard
108c831230 Update for 2019.02-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-23 15:06:12 +01:00
Peter Korsgaard
23a2885333 Update for 2019.02-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-13 09:03:54 +01:00
Carlos Santos
0c96bda11e Makefile: allow rootfs overlays to override symbolic links
Since commit 0db34529f4 we use rsync with the --keep-dirlinks option to
prevent overlays from accidentally overwriding /{usr,bin,sbin,lib} links
when BR2_ROOTFS_MERGED_USR option is enabled. Unfortunately this also
prevents replacing a symlink by a directory on purpose (e.g. /var/log,
to persist system logs).

Steps to reproduce:

- enable BR2_ROOTFS_MERGED_USR and BR2_PACKAGE_SKELETON_INIT_SYSV
- mkdir some_path/rootfs-overlay/var/log
- enable BR2_ROOTFS_OVERLAY="some_path/rootfs-overlay"
- run 'make'
- 'target/var/log' is still a symlink to '../tmp', not a directory

The --keep-dirlinks option can be dropped, since we run sanity checks
on overlays. Now the rsync invocation is identical to the SYSTEM_RSYNC
logic we have in system/system.mk, so use that variable.

Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 17:11:02 +01:00
Carlos Santos
dc7c6487cf Makefile: check rootfs overlays with BR2_ROOTFS_MERGED_USR enabled
Add a step to target-finalize that checks each rootfs overlay, following
the criteria established for custom skeletons and using the same script
uesd by skeleton-custom.mk.

Add a paragraph to the documentation clarifying that rootfs overlays
don't need to contain /bin, /lib or /sbin and must not contain them when
BR2_ROOTFS_MERGED_USR is enabled.

Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 17:09:28 +01:00
Joel Carlson
8fed162987 core/sdk: don't mangle symlinks with '.' or '..' at start
The current transform changes any '.' at the start of a filename to
$(BR2_SDK_PREFIX). This also applies to the target of a symlink, when
it is relative.

We thus might end up with something like:
    $(BR2_SDK_PREFIX)/bin/aarch64-linux-gnu-ar ->
    $(BR2_SDK_PREFIX)./opt/ext-toolchain/bin/aarch64-linux-gnu-ar

when it should be:
    $(BR2_SDK_PREFIX)/bin/aarch64-linux-gnu-ar ->
    ../opt/ext-toolchain/bin/aarch64-linux-gnu-ar

We fix that by making sure we always remove a known prefix, i.e. we
remove the path to host dir. The obvious solution would be to cd into
$(HOST_DIR)/.. , then tar ./host/ and finally use a --transfrom pattern
as 's,^\./$(notdir $(HOST_DIR)),$(BR2_SDK_PREFIX)'.

Since $(HOST_DIR) can point to a user-supplied location, we don't know
very well how the pattern may patch.

Instead, we cd into / and tar the full path to $(HOST_DIR).

Since tar removes any leading '/', it would spurr a warning message,
which is annoying. So we explicitly remove the leading '/' from
$(HOST_DIR) when we tar it.

Finally, we transform all filenames to replace a leading $(HOST_DIR)
(without a leading /) to the prefix to use.

Signed-off-by: Joel Carlson <JoelsonCarl@gmail.com>
[yann.morin.1998@free.fr:
  - use a single transform pattern
  - use full HOST_DIR path as pattern to replace
  - update commit log accordingly
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 16:30:14 +01:00
Thomas De Schampheleire
d3e535a839 Makefile: unexport 'PLATFORM' and 'OS' environment variables
Some package builds may fail when environment variables are present with the
same names as make variables in a package. This is a bigger problem for
environment variables with generic names, like 'PLATFORM' and 'OS'.

'PLATFORM' is for example a problem for host-acl.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 15:22:30 +01:00
John Keeping
e5e0f637ab Makefile: respect strip exclusions for special libraries
ld-*.so and libpthread*.so* are not stripped in the same way as other
binaries because some applications need symbols in these libraries in
order to operate correctly.

However, the special handling for these binaries ignores the usual
BR2_STRIP_EXCLUDE_* rules so it is not possible to build an image which
has debugging symbols in these binaries.

Pull out the common find functionality so that we can build two find
commands that re-use the common exclusion rules.

Fix-suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: John Keeping <john@metanate.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-02-04 16:56:01 +01:00
Vivien Didelot
45f0395971 Makefile: add update-defconfig target
For symmetry with the Kconfig-based packages offering comprehensive
targets like linux-update-defconfig, barebox-update-defconfig and so
on, add a new top level update-defconfig target to run savedefconfig.

Signed-off-by: Vivien Didelot <vivien.didelot@gmail.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-04 14:35:17 +01:00
Thomas Petazzoni
a86b626b5b Makefile: move definition of TARGET_DIR inside .config condition
In a follow-up commit introducing per-package directory support, we
will need to define TARGET_DIR in a different way depending on the
value of a Config.in option. To make this possible, the definition of
TARGET_DIR should be moved inside the BR2_HAVE_DOT_CONFIG condition.

We have verified that $(TARGET_DIR) is only used within the
BR2_HAVE_DOT_CONFIG condition. Outside of this condition, such as in
the "clean" target, $(BASE_TARGET_DIR) is used.

Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-17 22:38:52 +01:00
Peter Korsgaard
8e928a8389 Makefile, manual, website: Bump copyright year
Happy 2019!

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-06 21:30:34 +01:00
Adam Duskett
bbf32a77ec utils/test-pkg: force checking dependencies
Currently, if a user runs "make" while specifying a specific package
(IE: make -p foo),  the Makefile logic skips checking to see if all the
dependencies are selected in the specified packages config file. This behavior
is useful to test simple packages which do not have "complex" dependencies.

However; if a developer uses test-pkg -p ${package_name} to check their package,
the package may pass all the checks, but would have otherwise failed with a
simple "make" because the developer may have failed to add a select line in
packages config file, even if there is a new dependency in the packages
Makefile.

Pass the environment variable "BR_FORCE_CHECK_DEPENDENCIES"  to the Makefile in
the test-pkg script,  and check it's value in the Makefile. If the value is
"YES" force checking for dependency issues.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-03 11:41:26 +01:00
Ricardo Martincoski
e7e30455ef Makefile: offload .gitlab-ci.yml generation
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
|    only:
|        - /-$CI_JOB_NAME$/
|        - /-\$CI_JOB_NAME$/
|        - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
|    only:
|        refs:
|            - branches
|            - tags
|        variables:
|            - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
|    only:
|        - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
|    <<: *runtime_test
|    only:
|        - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.

So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.

Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.

Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.

While moving the commands to the script:
 - mimic the behavior of the previous make target and fail on any
   command that fails, by using 'set -e';
 - break the original lines in one command per line, making the diff for
   any patch to be applied to this file to look nicer;
 - keep the script as simple as possible, without functions, just a
   script that executes from the top to bottom;
 - do not perform validations on the input parameters, any command that
   fails already makes the script to fail;
 - do not add an usage message, the script is not intended to be called
   directly.

This patch does not change functionality.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-09 21:30:24 +01:00
Peter Korsgaard
13c43455a0 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-02 08:16:10 +01:00
Peter Korsgaard
9019189bd1 Kickoff 2019.02 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-01 23:36:34 +01:00
Peter Korsgaard
9089a9ff30 Update for 2018.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-01 23:06:49 +01:00
Peter Korsgaard
0031f52190 Update for 2018.11-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-30 13:27:09 +01:00
Thomas Petazzoni
beef2b4ab8 Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR
In commit 7e9870ce32 ("core: introduce
intermediate BASE_TARGET_DIR variable"), the definition of
TARGET_DIR_WARNING_FILE was changed to use $(BASE_TARGET_DIR) instead
of $(TARGET_DIR).

However, this change is incompatible with per-package directories, and
is in fact not needed.

With per-package directories, using $(BASE_TARGET_DIR) means that
TARGET_DIR_WARNING_FILE is
output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM. Due to this, when
skeleton-init-common or skeleton-custom attempt to install it, it
fails, because it should be installed to their package per-package
target directory, and not the global output/target directory that doesn't
exist yet. The failure looks like this:

/usr/bin/install -m 0644 support/misc/target-dir-warning.txt /home/thomas/projets/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
/usr/bin/install: cannot create regular file '/home/thomas/projets/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM': No such file or directory
make[1]: *** [package/pkg-generic.mk:336: /home/thomas/projets/buildroot/output/build/skeleton-init-common/.stamp_target_installed] Error 1

TARGET_DIR_WARNING_FILE is used in three places:

 - In skeleton-custom.mk and skeleton-init-common.mk, where as
   explained above, using $(TARGET_DIR) fixes the use of
   $(TARGET_DIR_WARNING_FILE) in the context of per-package target
   directories.

 - In fs/common.mk, where it is used as argument to $(notdir ...) to
   retrieve just the name of the warning file. So in this case, we
   really don't care about the path of the file, just its name.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:11:19 +01:00
Thomas Petazzoni
052fec0c08 Makefile: move .NOTPARALLEL statement after including .config file
In a follow-up commit, we will make the .NOTPARALLEL statement
conditional on a Config.in option, so we need to move it further down.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:10:57 +01:00
Thomas Petazzoni
d0f4f95e39 Makefile: rework main directory creation logic
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.

For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.

This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.

Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.

Thanks to those changes, the global "dirs" target is entirely removed.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:09:46 +01:00
Thomas Petazzoni
11e8c900ff Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use
As we are going to move to per-package SDK, the location of CCACHE and
therefore the definitions of HOSTCC and HOSTCXX need to be evaluated
at the time of use and not at the time of assignment. Indeed, the
value of HOST_DIR changes from one package to the other.

Therefore, we need to change from := to =.

In addition, while doing A := $(something) $(A) is possible, doing A =
$(something) $(A) is not legal. So, instead of defining HOSTCC in
terms of the current HOSTCC variable, we re-use HOSTCC_NOCCACHE
instead.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:08:13 +01:00
Peter Korsgaard
bc89c1a834 Update for 2018.11-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-21 08:44:25 +01:00
Peter Korsgaard
419fc6abca Update for 2018.11-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-09 22:56:48 +01:00
Yann E. MORIN
3950e69dad core: support host gcc of the future
When we do a release, we know only of a set of gcc versions that the
host may have. But in the future, distributions with newer gcc versions
may show up.

Currently, we do not recognise those versions, and thus we do as if they
were older than the oldest we know of. This means that a set of packages
become unselectable, when they should be.

We fix that by capping the detected version to the highest we know of.

Reported-by: gargar_ on IRC
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-23 11:43:35 +02:00
Arnout Vandecappelle (Essensium/Mind)
7099476882 Makefile: .gitlab-ci.yml: fail when listing tests fail
To update the .gitlab-ci.yml file, we run run-tests -l to list all the
tests and post-process the output in a format suitable for
.gitlab-ci.yml. However, in a pipeline, it is the last command that
gives the return value. In addition, we have to redirect stderr of
run-tests -l because nose2 prints the tests on stderr, not stdout. Thus,
when run-tests -l fails, the update of .gitlab-ci.yml silently succeeds
but no tests are included in the .gitlab-ci.yml.

To fix this, set the pipefail option. This is bash-specific, but our
Makefile ascertains that we are running with bash as the shell (if bash
is available, but if it is not, dependencies.sh will error out). The
error message is still invisible, but at least make will fail.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 23:33:04 +02:00
Michal Sojka
e33ea1c9a2 core/legal-info: Add package dependencies with licenses to the manifest
This adds one column to the legal-info manifest table. It contains the
dependencies of the given package and their licenses. This information
is useful when assessing license compatibility of the packages and
their libraries.

An example of the content of the new column for the MPD package is
shown below:

    "alsa-lib [LGPL-2.1+ (library), GPL-2.0+ (aserver)] boost
    [BSL-1.0] libid3tag [GPL-2.0+] libmad [GPL-2.0+] libogg
    [BSD-3-Clause] libvorbis [BSD-3-Clause] libzlib [Zlib]
    skeleton-init-common [unknown] skeleton-init-sysv [unknown] sqlite
    [Public domain] toolchain-external-linaro-arm [unknown]"

[Credits to Yann E. MORIN <yann.morin.1998@free.fr> for suggesting a
few simplifications.]

Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 19:13:20 +02:00
Michal Sojka
8f14901043 core/legal-info: Change order of legal-manifest parameters
The last parameter {HOST|TARGET} is now first. With this change,
adding new columns to the legal manifest file (as in the next commit)
will be slightly easier to review.

Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 19:12:31 +02:00