Commit fd7312940a (Makefile: add new missing-cpe target) added the
rule to generate a set of files to update the NVD.
For an in-tree build, 'make clean' remove the output directory, so
those files are removed. But for an out-of-tree build, the output
directory is not removed, so those files still linger around after a
clean.
Explicitly remove them on clean, to cater for both cases.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Matthew Weber <matthew.weber@collins.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 7d17ae2acf (.flake8: fix check for 80/132 columns) introduced a
difference in how flake8 behaves between the automatic checks done in
the CI, where the maximum line length is 132, and the local checks,
where the maximum line length is 80.
The rationale at the time was that we recommend 80 char lines, but that
we accept 132 when it makes sense for readability.
However, this is very annoying when running flake8 locally, because of
two reasons:
1. human reviews on python scripts have not been as thorough as we did
expect; indeed, we've let a lot of long lines slip through; this
causes a lot of spurious failures that hide away the actual errors;
2. when hacking on a python script, the issues reported will not be
caused by the current changes, so the many reported failures
actually hide away the newly introduced issues.
Additionally, our 'make check-flake8' rule already enforces the 132-char
limit, and the issues reported are different than when manually running
flake8 on individual files.
Furthermore, the readability rationale for the 80-char limit is
definitely shattered by the mere rationale of allowing 132-char limit
for... readability...
We've arrived to a point where this separation is causing our checks
around flake8 to become mostly unusable and useless, as they do not
report meaningful issues, and people are no longer paying attention, and
this has caused actual issues to be introduced.
Finally, terminal emulators of today have long lifted the 80-char limit,
and are more than capable of displaying 132-char wide lines.
Switch back to using a 132-char limit.
This reverts commit 7d17ae2acf.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Now that pkg-stats is not just a maintainer-oriented tool, but a tool
generally useful to users, introduce a make target to run
pkg-stats. Of course, it is run with the newly introduced -c option,
which produces a pkg-stats output for just the selection of packages
of the currently defined configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Although BR2_DL_DIR is indeed a site-local setting, which does not
actually define the target system, we've had it in the tree for a
long time now, and people have been depending on it for a variety
of use-cases.
Furthermore, BR2_DL_DIR is far from the only such site-local setting,
BR2_CCACHE_DIR springs to mind, and in the less-obvious category, we
can also find BR2_JLEVEL, but also BR2_WGET, BR2_SVN, BR2_GIT et al.
as they may be tweaked to set the timeout, number of retries or so on
to work around stupid proxies. But of course, the most local site-local
setting is probably BR2_PACKAGE_OVERRIDE_FILE, with its default value
being explicitly just 'local.mk'.
Ideally, we would like to have a clear separation between the
configuration that actually defines the target system on one hand,
and the site-local settings that drive and control how the build is
performed, on the other hand. This is by far a much bigger endeavour
than just dropping BR2_DL_DIR from the saved defconfig.
This reverts commit 36edacce9c (adapted
to keep the fix from 1a7873ec98).
Closes: #13291
Note: thanks to Thomas; some phrasing above was borrowed from a
discussion with him.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Lance Fredrickson <lancethepants@gmail.com>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
When we savedefconfig, we remove BR2_DEFCONFIG (f71a621d91, savedefconfig:
Remove BR2_DEFCONFIG from saved defconfig file) and BR2_DL_DIR (36edacce9c,
Makefile: exclude BR2_DL_DIR from savedefconfig), because their meaning
is only valid locally.
However, we were not careful to really match the exact variables, so we
could match arbitrary options.
For example, these config options would all be dropped:
BR2_DEFCONFIG="toto"
BR2_DL_DIR="titi"
BR2_PACKAGE_SABR2_DEFCONFIG="tutu"
BR2_PACKAGE_SABR2_DL_DIR=y
While the first two are indeed the ones we want to drop, the last two
are options (whatever their meaning or how poorly named they are) of the
hypothetical 'sabr2' package, and we want to keep those in a defconfig.
When cleaning the just-saved defconfig, be sure to anchor the patterns to
the beginning of the line.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Herve Codina <Herve.CODINA@celad.com>
Acked-by: Herve Codina <Herve.CODINA@celad.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running "make savedefconfig" with BR2_DL_DIR set also saves that
variable, which is only useful in local context.
Signed-off-by: Sven Oliver Moll <buildroot@svol.li>
[yann.morin.1998@free.fr: wrap line]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: drop supperfluous depends on s390x in choice]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Move the code to run check-flake8 into the Makefile, like we have for
check-package, so that it is easy to run locally (and not wait for
someone to report a failure from their Gitlab pipelines).
Compared to the existing check from gitlab-ci.yml, the Makefile check
differs in this respect:
- don't explicitly find *.py files: they are supposed to also be found
as a result of running 'file' on them;
- use git ls-tree instead of find: this is supopsedly faster as it
uses the index rather than readdir();
- don't output the count of warnings or errors: the output is a single
integer, which is confusing when there are errors, and even more so
when there are no, when it is simply '0';
- don't sort: the output is already stable and independent from the
locale;
- don't report the number of processed files: this information is
rather useless, and getting a hold of it would be more challenging
in this new code.
Note: ideally, we would want to use --null, --zero, or similar options,
with utilities that generates or parses a files listing. While git
ls-tree and xargs do support it, it becomes a little bit tricky to use
the --print0 option of file, and then grep in that output (it is not
undoable, but would requires replacing grep+cut with some sed trickery).
Since we do not expect our scripts names to contain funky chars (like
\n or a colon), we just hand-wave away that issue (and the old code was
doing the same assumption too).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As 18f6c26118 just did to silence the file lists commands, switch to
using $(Q) instead of a plain @, to silence the commands.
Using $(Q) will allow to debug the commands with V=1.
We keep @ for the calls to MESSAGE, though.
The commands that are not currently silenced are left as-is, and they
can be converted to being silent in a followup patch, if need be,
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since commit 0e2be4db8a
("package/pkg-generic: make file list logic parallel build
compatible"), the commands executed at the every end of the build
to assemble the list of files installed by the different packages
are visible in the make output. They are quite noisy, and clutter
the output.
The other commands in target-finalize are also hidden using "@",
so we should also do the same for those commands. But that hurts
debuggability, so we use $(Q) (the existing '@'s can be changed
in a followup patch).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use '$(Q)', not '@']
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
BR2_VERSION_FULL is currently defined as follows:
BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.
The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.
If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).
This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.
The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.
Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:
$ make print-version
2020.08-git-00758-gc351877a6e
So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:
$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e
This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.
So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.
In the above situation, with this commit applied, we get:
$ make print-version
dummy-tag-6-g6258cdddeb
(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)
Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since Gitlab 12.9, Gitlab allow to trigger child pipeline with generated configuration file.
See: https://gitlab.com/gitlab-org/gitlab/-/issues/35632
This allow us to stop updating the .gitlab-ci.yml file when a
new defconfig is added to Buildroot.
Remove check-gitlab-ci.yml job since it is now uneeded.
Remove .gitlab-ci.yml make target.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[ann.morin.1998@free.fr: manual: no longer needed to update at all]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The rule to create the staging symlink has it depend on BASE_DIR, and
the symlink is created in BASE_DIR, which means that when the symlink
is created, BASE_DIR is updated, and thus made more recent than the
symlink itself.
As a consequence, every time one runs 'make', the symlink will be older
than BASE_DIR, and so will be re-created.
Ditto for the host symlink when the user has elected to have an
out-of-tree host dir.
Fix that by changing to using an order-only dependency.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we delete /usr/share/bash-completion when bash is not enabled.
We need to delete /etc/bash_completion.d too. For example, the jo package
installs files there:
/etc/bash_completion.d/jo.bash
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some toolchains, like the Linaro gcc7 toolchains, now install libstdc++ debug
library symbols to /lib/debug, which can be as large as the library itself.
This commit removes the extra debug content if debugging is not enabled.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we prepare the release, we generate the manual in various formats,
so that it can be consulted locally without needing the miriads of tools
needed to generate it.
However, this creates the temporary .br2-external.* files in the output
directory, and those end up in the release tarball.
This is not a problem in practice, but is not clean.
Run 'distclean' in the output directory, to get rid of everything but
the generated documentation.
Reported-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Post-build scripts may want to do something based on the list of files
installed by a package. However, since commit
0e2be4db8a the final packages-file-lists.txt
file is only created _after_ the post-build scripts.
Move the assembly of the file lists upwards, before the post-build scripts.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In very limited configurations, it is possible to have a case where no
.files-list-staging.txt files are created. In this case:
cat $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) > \
$(BUILD_DIR)/packages-file-list-staging.txt
becomes:
cat > \
$(BUILD_DIR)/packages-file-list-staging.txt
which of course makes the build hang.. forever.
So we fix this by checking the list is not empty. To keep the code
readable, we introduce an intermediate variable to store the list of
these files.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: always create the file, even if empty]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
With BR2_PER_PACKAGE_DIRECTORIES=y, we have the following code in the
main Makefile:
target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
@$(call MESSAGE,"Finalizing target directory")
$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
The per-package-rsync call creates the global $(TARGET_DIR) from the
per-package $(TARGET_DIR). Then, we call the TARGET_FINALIZE_HOOKS.
One of the TARGET_FINALIZE_HOOKS, PURGE_LOCALES, remove locales that
are not desired by the user. It does so using a loop with the
$(wildcard ...) function.
However, the $(wildcard ...) function is expanded at the moment the
rule is evaluated. And with per-package directory, at the time the
rule is evaluated, the global $(TARGET_DIR) is empty, so $(wildcard
...) will return nothing. It is indeed only after the call to
per-package-rsync that the TARGET_DIR will be populated.
This commit fixes that by moving away from $(wildcard ...) and use a
shell test instead, since we are anyway in big block of shell code.
With this, locales are properly purged again when
BR2_PER_PACKAGE_DIRECTORIES=y.
Fixes: c4e6d5c8be ("core: implement per-package SDK and target")
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- make the style look like the code around (no space in front of ;)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This provides generic functions for Qt5 qmake based packages. It will
make it possible to remove lots of redefinition of
QT5_xxx_{CONFIGURE|BUILD|INSTALL_STAGING}_CMDS. Additionally it
provides a generic target install method which will make most of the
package specific commands obsolete.
This is done by re-running the install step of the qmake generated
Makefile with the package build directory prepended (to the
staging/host path). Even though this does create lengthy pathes it
allows for easy separation of the staging files from the host destined
files by just omitting the resulting BUILD_DIR+HOST_DIR path from the
following rsync call to the real target folder. The cleanup of many
files we dont want in target is deferred to the target-finalize
step. In addition to what's being removed already, we also have to
cleanup some Qt5 specific files (prl) and the documentation directory.
This approach was chosen over copying all files recorded in the pkg-files-list
after some discussion which Thomas Petazzoni summed up:
"We don't yet use pkg-files-list really as part of the build
process anywhere, I feel a bit more comfortable at this point with what
Andreas is proposing."
Thanks to this infrastructure, it will be possible to get rid of the
many conditional install commands because qmake already takes care of
this when generating the Makefile install targets with the given or
autodetected configure options of each package.
However, custom install steps may have to remain in cases where a
particular Buildroot option has no corresponding setting in the
packages configuration options.
Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Several users of rolling-release distributions have been reporting on
IRC that Buildroot is broken now that they have switched to the newly
released make 4.3.
It turns out that the constructs we use to generated and include the
internal br2-external related fragments is no longer working with
make-4.3.
Indeed, an upstream bug report [0] seems to imply that it so far was
working by chance. There has been no further feedback, whether this is
really considered a fix for a previous ill-defined behaviour, or an
actual regression...
In the meantime, we add a workaround, suggested in that same bug report,
that fixes the issue for make 4.3, and that should not break on older
make versions either (verified on all relevant versions: from 3.81,
3.82, 4.0, 4.1, and 4.2).
[0] https://savannah.gnu.org/bugs/?57676
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Mircea Gliga <mgliga@bitdefender.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The comment "Check files that are touched by more than one package"
was previously located right before the calls to the check-uniq-files
script. However, this script and the logic calling it have been
removed in commit 2496189a42 ("core:
drop check-uniq-files"), so the comment no longer makes any sense:
let's drop it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.
This works well when packages are built sequentially, but badly fails
when using top-level parallel build.
More specifically, top-level parallel build can fail with:
comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory
Because that file has been removed concurrently by the build process
of another package.
This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.
At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.
There are however some differences:
(1) The files are no longer ordered in build order, but by alphabetic
ordering of packages. Indeed, "build order" no longer makes any
sense in the context of top-level parallel build.
(2) Some files which were incorrectly tracked are no longer
tracked. For example, the toolchain package is a target package,
but it installs files in $(HOST_DIR). In the previous logic, the
files installed by the toolchain package in $(HOST_DIR) were
incorrectly affected to the next host package that was installed
after the toolchain package. With our new logic, those files are
no longer tracked at all. To fix this, we would have to change
the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
installation steps, not just for the install-host, install-target
and install-staging steps respecitively. But the result was
already incorrect anyway, and therefore this should be fixed
separately.
Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).
Fixes:
http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 509db3b88a added calls to (parts of) the instrumentation steps.
However, those calls are echoed, unlike the other places where we call
them (in the package infra).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Acked-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Create the staging symlink the same way as the host symlink. This means
using a make dependency rather than recreating it every time.
In coreutils versions below 8.27, re-creation of symbolic links was not
atomic. This means that there is a period in time where the existing link is
removed, before the new one is created. In coreutils 8.27 this was fixed,
see [1]. Note that CentOS 7 ships with coreutils 8.22.
In the following scenario, this is a problem:
- an application is compiled using the sysroot prepared by Buildroot and
links against Xenomai userspace libraries, but its build process is steered
from outside of Buildroot
- to know the correct flags, the application makefile uses the 'xeno-config'
file to request them, and passes DESTDIR=/buildroot/output/staging
- the xeno-config responds with flags based on the path
'/buildroot/output/staging/...'
- while the application build is ongoing, a 'make' happens in Buildroot,
causing the 'staging' symlink to be recreated (even though it already
existed)
- when exactly at this time, the application calls the compiler with -I
flags pointing to output/staging, the build fails with:
-I/buildroot/output/staging/usr/include/xenomai/mercury: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai/xenomai: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai/psos: Error: ^ is not a directory
Failed: ** ^ *
Work around this problem by only creating the staging symlink once, similar
to how the host symlink (if any) is created.
See also commit d0f4f95e39 which changed the
way these symlinks are made. The reasoning in this commit is to move away
from the 'dirs' target.
[1] 376967889e
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The package instrumentation step 'step_pkg_size' is populating the files:
output/build/packages-file-list.txt
output/build/packages-file-list-staging.txt
output/build/packages-file-list-host.txt
by comparing the list of files before and after installation of a package,
with some clever tricks to detect changes to existing files etc.
As an optimization, instead of gathering this list before and after each
package, where the 'after-state' of one package is the same as the
'before-state' of the next package, only the 'after-state' is used and
is shared between packages.
This works fine, except at the end of the build, as explained next.
In the target-finalize step, many files will be touched. For example, files
like /etc/hosts, /etc/os-release, but also all object files that are
stripped, and all files touched by post-build scripts or created by rootfs
overlays. This means that the 'after-state' of the last package does not
reflect the actual situation after target-finalize is run.
For a single complete build this poses no problem. But, if one incrementally
rebuilds a package after the initial build, e.g. with 'make foo-rebuild',
then all changes that happened in target-finalize at the end of the initial
build (the 'after-state' of the last package built) will be detected as
changes caused by the rebuild of package foo. As a result, all these files
will incorrectly be treated as 'owned' by package foo.
Correct this situation by capturing a new state at the end of
target-finalize, so that the 'before-state' of an incremental build will be
correct.
Note: the reasoning above talks about packages-file-list.txt and
target-finalize, but also applies to
packages-file-list-staging.txt/staging-finalize and
packages-file-list-host.txt/host-finalize.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With per-package folder support, top-level parallel build becomes
safe, so we can enclose the .NOTPARALLEL statement in a
!BR2_PER_PACKAGE_DIRECTORIES condition.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Back a few years ago, when we were starting to think about top-level
parallel build, we were not sure how to deal with packages that
installed the same files, so we wanted to catch the situation to assess
how prevalent that was, before we decided what to do and how to address
it.
However, the trend nowadays is that packages will install in a
per-package target/ (and staging/ and host/), and the final directories
will be assembled in a reproducible (alphabetical) order, so if two
packages install the same file, the last one will win (as is currently
the case).
Besides, check-uniq-files reports loads of spurious errors when packages
get reinstalled (e.g. during development).
Finally, check-uniq-files is the only script called during the build,
that is written in python.
So, get rid of check-uniq-files.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, we group packages that contribute less then 1%, into the
"Other" category.
However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.
Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.
Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
- remove empty base class definition from Config;
- use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When no filesystem is enabled, the $BINARIES_DIR is not created. Yet,
the post-image scripts are still run. When those want to generate an
image in there, they may fail as the dirctory does not exist (it did
exist before we started applying preparatory changes for top-level
parallel build, so scripts got to rely on that assumption).
Do in target-post-image as we do in the sdk rule: create the directory
before calling the scripts.
Signed-off-by: Brent Generous <bgenerous@impinj.com>
[yann.morin.1998@free.fr:
- create the directory before calling the scripts
- don't drop the creation in the sdk rule
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This rule was added back in 9429e7b698 (core: introduce an intermediate
rule before the configurators) when the kconfig-side br2-external file
was generated separately from the Makefile-side one.
Now that they are generated together very early in the Makefile, we no
longer need this intermediate rule. Drop it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
[Peter: also drop outdated reference in the manual]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.
But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.
And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.
Make it so.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that all the br2-external generated files are named after the same
pattern, it gets easier to remove them all using a glob.
Furthermore, we're on the verge of introducing more such generated
files, so removing them at one fell swoop will be simpler too.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that the two (all of them!) br2-external related files are generated
in the same location, it makes sense they are named after the same
pattern.
When initial support for (then single) br2-external trees was added back
in a4239f7fd1 (core: introduce the BR2_EXTERNAL variable), it was not
clear-cut why that file was not named with a br2 prefix.
So rename it now.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, that file is generated rather late in the configuration
process, so BUILD_DIR is known (and exists) by then.
We're soon to generate that file much earlier, at a point where
BUILD_DIR is not yet known, so we have two options:
1- declare BUILD_DIR earlier;
2- generate the file in an already-known location.
We go with the second solution, as we're already generating a
br2-external related file in BASE_DIR, so we can as well generate all
br2-external files in the same place.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We export GZIP = -n so that GZIP does not record original
name and timestamps. However..
GZIP environment variable is deprecated and soon will not be
supported in future GZIP versions. GZIP suggests the use of a
wrapper to pass options globally but it might be difficult to
implement in Buildroot. For now, we don't export the variable
and fix reproducibility issues per package as they show up in
Autobuilder.
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.
show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.
Beside, show-dependency-tree has never been part of a release.
Drop it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.
Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.
Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:
$ make {foo,bar}-show-info
{ "foo": { foo stuff } }
{ "bar": { bar stuff } }
However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):
$ make {foo,bar}-show-info |jq -s . -
[
{ "foo": { foo stuff } },
{ "bar": { bar stuff } }
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Commit 15cb98769e (release: remove manual build files from release
tarballs) tried to remove the temporary files from the manual build from the
release tarball, but manual-clean only removes build/docs/manual and leaves
build/docs in the tarball.
Instead use 'make clean' to completely remove the build directory from the
tarball.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.
Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.
However, make being make, already builds the whole dependency tree
information, so we can just ask for it.
Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:
foo: target 1.2.3
foo -> bar host-meh
bar: target virtual
bar -> buz
buz: target 2.3.4
buz ->
host-meh: host virtual
host-meh -> host-bleark
host-bleark: host 3.4.5
host-bleark ->
rootfs-meh: host
rootfs-meh -> host-bleark
To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.
For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):
$ time make graph-depends
real 0m27.344s
$ time make show-dependency-tree
real 0m3.848s
>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):
$ time make graph-depends
real 1m56.459s
$ time make show-dependency-tree
real 0m5.748s
There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Using 'make printvars' for printing all variables is not very useful.
E.g. all macros will output some bogus value. In addition, the same can
be achieved with 'make -p'.
We can simply remove the condition on $(VARS). If VARS is not set, the
filter expression will be empty which matches nothing, so nothing is
printed.
Note that the old behaviour can still be achieved with:
make printvars VARS=%
Update the 'make help' text to match the new behaviour.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since commit 0db34529f4 we use rsync with the --keep-dirlinks option to
prevent overlays from accidentally overwriding /{usr,bin,sbin,lib} links
when BR2_ROOTFS_MERGED_USR option is enabled. Unfortunately this also
prevents replacing a symlink by a directory on purpose (e.g. /var/log,
to persist system logs).
Steps to reproduce:
- enable BR2_ROOTFS_MERGED_USR and BR2_PACKAGE_SKELETON_INIT_SYSV
- mkdir some_path/rootfs-overlay/var/log
- enable BR2_ROOTFS_OVERLAY="some_path/rootfs-overlay"
- run 'make'
- 'target/var/log' is still a symlink to '../tmp', not a directory
The --keep-dirlinks option can be dropped, since we run sanity checks
on overlays. Now the rsync invocation is identical to the SYSTEM_RSYNC
logic we have in system/system.mk, so use that variable.
Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a step to target-finalize that checks each rootfs overlay, following
the criteria established for custom skeletons and using the same script
uesd by skeleton-custom.mk.
Add a paragraph to the documentation clarifying that rootfs overlays
don't need to contain /bin, /lib or /sbin and must not contain them when
BR2_ROOTFS_MERGED_USR is enabled.
Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The current transform changes any '.' at the start of a filename to
$(BR2_SDK_PREFIX). This also applies to the target of a symlink, when
it is relative.
We thus might end up with something like:
$(BR2_SDK_PREFIX)/bin/aarch64-linux-gnu-ar ->
$(BR2_SDK_PREFIX)./opt/ext-toolchain/bin/aarch64-linux-gnu-ar
when it should be:
$(BR2_SDK_PREFIX)/bin/aarch64-linux-gnu-ar ->
../opt/ext-toolchain/bin/aarch64-linux-gnu-ar
We fix that by making sure we always remove a known prefix, i.e. we
remove the path to host dir. The obvious solution would be to cd into
$(HOST_DIR)/.. , then tar ./host/ and finally use a --transfrom pattern
as 's,^\./$(notdir $(HOST_DIR)),$(BR2_SDK_PREFIX)'.
Since $(HOST_DIR) can point to a user-supplied location, we don't know
very well how the pattern may patch.
Instead, we cd into / and tar the full path to $(HOST_DIR).
Since tar removes any leading '/', it would spurr a warning message,
which is annoying. So we explicitly remove the leading '/' from
$(HOST_DIR) when we tar it.
Finally, we transform all filenames to replace a leading $(HOST_DIR)
(without a leading /) to the prefix to use.
Signed-off-by: Joel Carlson <JoelsonCarl@gmail.com>
[yann.morin.1998@free.fr:
- use a single transform pattern
- use full HOST_DIR path as pattern to replace
- update commit log accordingly
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Some package builds may fail when environment variables are present with the
same names as make variables in a package. This is a bigger problem for
environment variables with generic names, like 'PLATFORM' and 'OS'.
'PLATFORM' is for example a problem for host-acl.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
ld-*.so and libpthread*.so* are not stripped in the same way as other
binaries because some applications need symbols in these libraries in
order to operate correctly.
However, the special handling for these binaries ignores the usual
BR2_STRIP_EXCLUDE_* rules so it is not possible to build an image which
has debugging symbols in these binaries.
Pull out the common find functionality so that we can build two find
commands that re-use the common exclusion rules.
Fix-suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: John Keeping <john@metanate.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For symmetry with the Kconfig-based packages offering comprehensive
targets like linux-update-defconfig, barebox-update-defconfig and so
on, add a new top level update-defconfig target to run savedefconfig.
Signed-off-by: Vivien Didelot <vivien.didelot@gmail.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In a follow-up commit introducing per-package directory support, we
will need to define TARGET_DIR in a different way depending on the
value of a Config.in option. To make this possible, the definition of
TARGET_DIR should be moved inside the BR2_HAVE_DOT_CONFIG condition.
We have verified that $(TARGET_DIR) is only used within the
BR2_HAVE_DOT_CONFIG condition. Outside of this condition, such as in
the "clean" target, $(BASE_TARGET_DIR) is used.
Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, if a user runs "make" while specifying a specific package
(IE: make -p foo), the Makefile logic skips checking to see if all the
dependencies are selected in the specified packages config file. This behavior
is useful to test simple packages which do not have "complex" dependencies.
However; if a developer uses test-pkg -p ${package_name} to check their package,
the package may pass all the checks, but would have otherwise failed with a
simple "make" because the developer may have failed to add a select line in
packages config file, even if there is a new dependency in the packages
Makefile.
Pass the environment variable "BR_FORCE_CHECK_DEPENDENCIES" to the Makefile in
the test-pkg script, and check it's value in the Makefile. If the value is
"YES" force checking for dependency issues.
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
| only:
| - /-$CI_JOB_NAME$/
| - /-\$CI_JOB_NAME$/
| - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
| only:
| refs:
| - branches
| - tags
| variables:
| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
| only:
| - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
| <<: *runtime_test
| only:
| - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.
So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.
Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.
Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.
While moving the commands to the script:
- mimic the behavior of the previous make target and fail on any
command that fails, by using 'set -e';
- break the original lines in one command per line, making the diff for
any patch to be applied to this file to look nicer;
- keep the script as simple as possible, without functions, just a
script that executes from the top to bottom;
- do not perform validations on the input parameters, any command that
fails already makes the script to fail;
- do not add an usage message, the script is not intended to be called
directly.
This patch does not change functionality.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In commit 7e9870ce32 ("core: introduce
intermediate BASE_TARGET_DIR variable"), the definition of
TARGET_DIR_WARNING_FILE was changed to use $(BASE_TARGET_DIR) instead
of $(TARGET_DIR).
However, this change is incompatible with per-package directories, and
is in fact not needed.
With per-package directories, using $(BASE_TARGET_DIR) means that
TARGET_DIR_WARNING_FILE is
output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM. Due to this, when
skeleton-init-common or skeleton-custom attempt to install it, it
fails, because it should be installed to their package per-package
target directory, and not the global output/target directory that doesn't
exist yet. The failure looks like this:
/usr/bin/install -m 0644 support/misc/target-dir-warning.txt /home/thomas/projets/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
/usr/bin/install: cannot create regular file '/home/thomas/projets/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM': No such file or directory
make[1]: *** [package/pkg-generic.mk:336: /home/thomas/projets/buildroot/output/build/skeleton-init-common/.stamp_target_installed] Error 1
TARGET_DIR_WARNING_FILE is used in three places:
- In skeleton-custom.mk and skeleton-init-common.mk, where as
explained above, using $(TARGET_DIR) fixes the use of
$(TARGET_DIR_WARNING_FILE) in the context of per-package target
directories.
- In fs/common.mk, where it is used as argument to $(notdir ...) to
retrieve just the name of the warning file. So in this case, we
really don't care about the path of the file, just its name.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In a follow-up commit, we will make the .NOTPARALLEL statement
conditional on a Config.in option, so we need to move it further down.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.
For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.
This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.
Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.
Thanks to those changes, the global "dirs" target is entirely removed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As we are going to move to per-package SDK, the location of CCACHE and
therefore the definitions of HOSTCC and HOSTCXX need to be evaluated
at the time of use and not at the time of assignment. Indeed, the
value of HOST_DIR changes from one package to the other.
Therefore, we need to change from := to =.
In addition, while doing A := $(something) $(A) is possible, doing A =
$(something) $(A) is not legal. So, instead of defining HOSTCC in
terms of the current HOSTCC variable, we re-use HOSTCC_NOCCACHE
instead.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When we do a release, we know only of a set of gcc versions that the
host may have. But in the future, distributions with newer gcc versions
may show up.
Currently, we do not recognise those versions, and thus we do as if they
were older than the oldest we know of. This means that a set of packages
become unselectable, when they should be.
We fix that by capping the detected version to the highest we know of.
Reported-by: gargar_ on IRC
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
To update the .gitlab-ci.yml file, we run run-tests -l to list all the
tests and post-process the output in a format suitable for
.gitlab-ci.yml. However, in a pipeline, it is the last command that
gives the return value. In addition, we have to redirect stderr of
run-tests -l because nose2 prints the tests on stderr, not stdout. Thus,
when run-tests -l fails, the update of .gitlab-ci.yml silently succeeds
but no tests are included in the .gitlab-ci.yml.
To fix this, set the pipefail option. This is bash-specific, but our
Makefile ascertains that we are running with bash as the shell (if bash
is available, but if it is not, dependencies.sh will error out). The
error message is still invisible, but at least make will fail.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This adds one column to the legal-info manifest table. It contains the
dependencies of the given package and their licenses. This information
is useful when assessing license compatibility of the packages and
their libraries.
An example of the content of the new column for the MPD package is
shown below:
"alsa-lib [LGPL-2.1+ (library), GPL-2.0+ (aserver)] boost
[BSL-1.0] libid3tag [GPL-2.0+] libmad [GPL-2.0+] libogg
[BSD-3-Clause] libvorbis [BSD-3-Clause] libzlib [Zlib]
skeleton-init-common [unknown] skeleton-init-sysv [unknown] sqlite
[Public domain] toolchain-external-linaro-arm [unknown]"
[Credits to Yann E. MORIN <yann.morin.1998@free.fr> for suggesting a
few simplifications.]
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The last parameter {HOST|TARGET} is now first. With this change,
adding new columns to the legal manifest file (as in the next commit)
will be slightly easier to review.
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
gcc does not build when the srcdir path contains a '@', because that
path is then substitued in a texi file as argument to an @include
directive. But then, the '@' in the path will start a command evaluation
of its own, thus breaking the build. For example, with a $(O) path set
to /home/ymorin/dev/buildroot/O/to@ti :
perl ../../gcc/../contrib/texi2pod.pl ../../gcc/doc/invoke.texi > gcc.pod
../../gcc/doc/invoke.texi:1678: unknown command `ti'
../../gcc/doc/invoke.texi:1678: @include: could not find /home/ymorin/dev/buildroot/O/to/build/host-gcc-initial-7.3.0/build/gcc/../../gcc/../libiberty/at-file.texi
[Peter: use findstring instead of subst/compare]
Reported-by: c32 on IRC
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This enables a riscv64 system to be built with a Buildroot generated
toolchain (gcc >= 7.x, binutils >= 2.30, glibc only).
This configuration has been used to successfully build a qemu-bootable
riscv-linux-4.15 kernel (https://github.com/riscv/riscv-linux.git).
Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
[Thomas:
- simplify arch.mk.riscv by directly setting GCC_TARGET_ARCH
- simplify glibc.mk changes by using GLIBC_CONF_ENV.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The BR2_GCC_TARGET_* configuration variables are copied to
corresponding GCC_TARGET_* variables which may then be optionally
modified or overwritten by architecture specific makefiles.
All makefiles must use the new GCC_TARGET_* variables instead
of the BR2_GCC_TARGET_* versions.
Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
[Thomas: simplify include of arch/arch.mk]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The underlying problem is that $(foreach V,1 2 3,) does not evaluate to
an empty string. It evaluates to " ", three empty strings separated by
whitespace.
A construct of this format, with a giant list in the foreach, is part of
the printvars command. This means that "@:$(foreach ....)", which is
intended to expand to a null command, in fact expands to "@: "
with a great deal of whitespace. Make chooses to execute this command
with:
execve("/bin/sh", ["/bin/sh", "-c", ": "]
But with far more whitespace. So much that it can exceed shell command
line length limits.
This solution is to move the foreach to another step in the recipe. The
"@:" is retained as the first line so the recipe is not Empty, which
would cause a change in make behavior when make builds the target. The
2nd line, all whitespace, will be skipped by make.
Signed-off-by: Trent Piepho <tpiepho@impinj.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, the wording in the manual instructs the user to generate a
tarball from "the contents of the +output/host+ directory".
This is pretty confusing, because taken literally, this would amount to
running a command like:
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when
extracted, it creates multiple files in the current directory.
What one really wants to do, is create a tarball of the host/ directory,
with something like:
tar cf my-sdk.tar -C output host/
However, this is not much better, because the top-most directory would
have a very common name, host/, which is pretty easy to get conflict
with when it gets extracted.
So, we fix that mess by giving the top-most directory a recognisable
name, based on the target tuple, which we also use as the name of the
archive (suffixed with the usual +.tar.gz+.) We offer the user the
possibility to override that default by specifying the +BR2_SDK_PREFIX+
variable on the command line.
Since this is an output file, we place it in the images/ directory.
As some users expressed a very strong feeling that they do not want to
generate a tarball at all, and that doing so would badly hurt their
workflows [1], we actually prepare the SDK as was previously done, but
under the new, intermediate rule 'prepare-sdk'. The existing 'sdk' rule
obviously depend on that before generating the tarball.
We choose to make the existing rule to generate the tarball, and
introduce a new rule to just prepare the SDK, rather than keep the
existing rule as-is and introduce a new one to generate the tarball,
because it makes sense to have the simplest rule do the correct thing,
leaving advanced, power users use the longest command. If someone
already had a wrapper that called 'sdk' and expected just the host
directory to be prepared, then this is not broken; it just takes a bit
longer (gzip is pretty fast).
Update the manual accordingly.
[0] https://en.wikipedia.org/wiki/Tar_(computing)#Tarbomb
[1] http://lists.busybox.net/pipermail/buildroot/2018-June/thread.html#223377
and some messages in the ensuing thread...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Wolfgang Grandegger <wg@grandegger.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Stefan Becker <chemobejk@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: "Yann E. MORIN" <<a href="mailto:yann.morin.1998@free.fr" target="_blank">yann.morin.1998@free.fr</a>><br>
Reviewed-by: Stefan Becker <chemobejk@gmail.com>
Signed-off-by: "Yann E. MORIN" <<a href="mailto:yann.morin.1998@free.fr" target="_blank">yann.morin.1998@free.fr</a>><br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The snippet of code that runs a check-package on all
.mk/.hash/Config.in files is currently only available within
.gitlab-ci.yml, and isn't immediately and easily usable by Buildroot
users. In order to simplify this, this commit introduces a top-level
"check-package" make target that implements the same logic. The
.gitlab-ci.yml file is changed to use "make check-package".
Since this target is oriented towards Buildroot developers, we
intentionally do not clutter the already noisy "make help" text with
this additional make target.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The help text says that BR2_DEFCONFIG will be used as input, but a
BR2_DEFCONFIG specified in the existing .config file will *not* be
used. So say explicitly that it must be specified on the command line.
Note that both "BR2_DEFCONFIG=... make defconfig" and
"make defconfig BR2_DEFCONFIG=..." will work.
While we're at it, add a semicolon to separate the two statements.
Note that this overflows the help text beyond 80 characters, but that
is already the case in many other lines.
Reported-by: Anisse Astier <anisse@astier.eu>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The systemd ConditionNeedsUpdate option is useful when offline updates
of the vendor operating system resources in /usr require updating of
/etc or /var on the next following boot.
Two examples of services making use of this option are
systemd-hwdb-update.service and systemd-sysusers.service.
ConditionNeedsUpdate=/etc will be true if the mtime of /etc/.updated
is older than the mtime of /usr. After services conditional on
ConditionNeedsUpdate have run, systemd-update-done.service will
synch the mtime of /usr to /etc/.updated so that the condition will
be false on subsequent boots.
For systems with writable /usr partitions where updates are done to
the running system, the update program will touch /usr as a final step.
But with Buildroot, where updates are often done by dumping a new
image onto the device, and where /usr is on a filesystem mounted
read-only, touching /usr as part of the update process is not practical.
Instead, it should be done a build time.
For testers, please note that systemd-update-done in v234 added a
regression where the mtime of /etc/.updated is set to the current time
instead of the mtime or /usr. This will be fixed in v239.
For more details, see:
http://0pointer.de/public/systemd-man/systemd.unit.htmlhttp://0pointer.de/public/systemd-man/systemd-update-done.service.html
Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The Blackfin architecture has for a long time been complicated to
maintain, with poor support in upstream binutils/gcc. As of April
2018, the Blackfin architecture has been dropped from the upstream
Linux kernel. Also, the Analog Device engineer who used to be in touch
with the Buildroot community also privately said we should drop the
support for this architecture, which Analog Devices is no longer
using, promoting and maintaining.
The BR2_BINFMT_FLAT_SEP_DATA option becomes unselectable, it will be
removed in a future commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For reproducible builds, SOURCE_DATE_EPOCH will be set to the git commit
date if it is not defined in the environment, but this was done by
explicitly using $(TOPDIR)/.git as the git repository, which would not
give the expected result if Buildroot had been put into a subdirectory
of another repository.
This commit removes that restriction, meaning that the default date will
now be the date of the git commit that contains Makefile, regardless of
what level above Makefile the repository is at. This works because the
current directory when the 'git log' command is executed will always be
the directory containing Makefile (it must be, since TOPDIR is set from
CURDIR).
In general this should be a sensible default, and in cases where a
different date is required SOURCE_DATE_EPOCH can be defined in the
environment before invoking make.
Signed-off-by: James Byrne <james.byrne@origamienergy.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
If SOURCE_DATE_EPOCH is not defined it was given a definition that
caused 'git log' to be executed each time the variable is referenced,
which is not very efficient given that the answer cannot change.
This commit moves the definition of BR2_VERSION_GIT_EPOCH after the
inclusion of Makefile.in (so that GIT is defined) and makes it a
simply expanded variable so that it is only evaluated once.
Signed-off-by: James Byrne <james.byrne@origamienergy.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds the support for <pkg>-show-recursive-depends and
<pkg>-show-recursive-rdepends which respectively show the list of all
dependencies or reverse dependencies for a given package. The existing
show-depends and show-rdepends only show the first-level dependencies,
while show-recursive-depends and show-recursive-rdepends show
recursively the dependencies.
It is worth mentioning that while show-recursive-depends really shows
all dependencies, show-recursive-rdepends is a bit limited because the
reverse dependencies of host packages are not properly accounted
for. But that's a limitation that already exists in show-rdepends, and
that cannot easily be solved.
Signed-off-by: George Redivo <george.redivo@datacom.ind.br>
[Thomas:
- split from the patch that was also changing graph-depends
- rename show-rrdepends to show-recursive-rdepends
- add show-recursive-depends
- don't create GRAPHS_DIR.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This was present in Yann's original patch, but got dropped when I rebased
commit 7e9870ce32 (core: introduce intermediate BASE_TARGET_DIR variable) to
fix the Makefile conflicts.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Each of the intermediate, per-rootfs target directories, as well as the
intermediate tarball, can take quite some place, and is mostly a
duplication of what's already in target/. The only delta, if any, would
be the tweaks made by the filesystem image generations, but those tweaks
are most probably only meaningful when seen as root.
We normally do not remove intermediate files, but those can be quite
large, and are not directly usable by, nor accessible to the user.
So, get rid of them once the filesystem has been generated.
This does not need to be done in fakeroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
... which for now still points to the base target directory, but this is
a step forward.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
