Now that pkg-stats is not just a maintainer-oriented tool, but a tool
generally useful to users, introduce a make target to run
pkg-stats. Of course, it is run with the newly introduced -c option,
which produces a pkg-stats output for just the selection of packages
of the currently defined configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Although BR2_DL_DIR is indeed a site-local setting, which does not
actually define the target system, we've had it in the tree for a
long time now, and people have been depending on it for a variety
of use-cases.
Furthermore, BR2_DL_DIR is far from the only such site-local setting,
BR2_CCACHE_DIR springs to mind, and in the less-obvious category, we
can also find BR2_JLEVEL, but also BR2_WGET, BR2_SVN, BR2_GIT et al.
as they may be tweaked to set the timeout, number of retries or so on
to work around stupid proxies. But of course, the most local site-local
setting is probably BR2_PACKAGE_OVERRIDE_FILE, with its default value
being explicitly just 'local.mk'.
Ideally, we would like to have a clear separation between the
configuration that actually defines the target system on one hand,
and the site-local settings that drive and control how the build is
performed, on the other hand. This is by far a much bigger endeavour
than just dropping BR2_DL_DIR from the saved defconfig.
This reverts commit 36edacce9c (adapted
to keep the fix from 1a7873ec98).
Closes: #13291
Note: thanks to Thomas; some phrasing above was borrowed from a
discussion with him.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Lance Fredrickson <lancethepants@gmail.com>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
When we savedefconfig, we remove BR2_DEFCONFIG (f71a621d91, savedefconfig:
Remove BR2_DEFCONFIG from saved defconfig file) and BR2_DL_DIR (36edacce9c,
Makefile: exclude BR2_DL_DIR from savedefconfig), because their meaning
is only valid locally.
However, we were not careful to really match the exact variables, so we
could match arbitrary options.
For example, these config options would all be dropped:
BR2_DEFCONFIG="toto"
BR2_DL_DIR="titi"
BR2_PACKAGE_SABR2_DEFCONFIG="tutu"
BR2_PACKAGE_SABR2_DL_DIR=y
While the first two are indeed the ones we want to drop, the last two
are options (whatever their meaning or how poorly named they are) of the
hypothetical 'sabr2' package, and we want to keep those in a defconfig.
When cleaning the just-saved defconfig, be sure to anchor the patterns to
the beginning of the line.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Herve Codina <Herve.CODINA@celad.com>
Acked-by: Herve Codina <Herve.CODINA@celad.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running "make savedefconfig" with BR2_DL_DIR set also saves that
variable, which is only useful in local context.
Signed-off-by: Sven Oliver Moll <buildroot@svol.li>
[yann.morin.1998@free.fr: wrap line]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: drop supperfluous depends on s390x in choice]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Move the code to run check-flake8 into the Makefile, like we have for
check-package, so that it is easy to run locally (and not wait for
someone to report a failure from their Gitlab pipelines).
Compared to the existing check from gitlab-ci.yml, the Makefile check
differs in this respect:
- don't explicitly find *.py files: they are supposed to also be found
as a result of running 'file' on them;
- use git ls-tree instead of find: this is supopsedly faster as it
uses the index rather than readdir();
- don't output the count of warnings or errors: the output is a single
integer, which is confusing when there are errors, and even more so
when there are no, when it is simply '0';
- don't sort: the output is already stable and independent from the
locale;
- don't report the number of processed files: this information is
rather useless, and getting a hold of it would be more challenging
in this new code.
Note: ideally, we would want to use --null, --zero, or similar options,
with utilities that generates or parses a files listing. While git
ls-tree and xargs do support it, it becomes a little bit tricky to use
the --print0 option of file, and then grep in that output (it is not
undoable, but would requires replacing grep+cut with some sed trickery).
Since we do not expect our scripts names to contain funky chars (like
\n or a colon), we just hand-wave away that issue (and the old code was
doing the same assumption too).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As 18f6c26118 just did to silence the file lists commands, switch to
using $(Q) instead of a plain @, to silence the commands.
Using $(Q) will allow to debug the commands with V=1.
We keep @ for the calls to MESSAGE, though.
The commands that are not currently silenced are left as-is, and they
can be converted to being silent in a followup patch, if need be,
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since commit 0e2be4db8a
("package/pkg-generic: make file list logic parallel build
compatible"), the commands executed at the every end of the build
to assemble the list of files installed by the different packages
are visible in the make output. They are quite noisy, and clutter
the output.
The other commands in target-finalize are also hidden using "@",
so we should also do the same for those commands. But that hurts
debuggability, so we use $(Q) (the existing '@'s can be changed
in a followup patch).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use '$(Q)', not '@']
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
BR2_VERSION_FULL is currently defined as follows:
BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.
The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.
If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).
This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.
The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.
Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:
$ make print-version
2020.08-git-00758-gc351877a6e
So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:
$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e
This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.
So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.
In the above situation, with this commit applied, we get:
$ make print-version
dummy-tag-6-g6258cdddeb
(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)
Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since Gitlab 12.9, Gitlab allow to trigger child pipeline with generated configuration file.
See: https://gitlab.com/gitlab-org/gitlab/-/issues/35632
This allow us to stop updating the .gitlab-ci.yml file when a
new defconfig is added to Buildroot.
Remove check-gitlab-ci.yml job since it is now uneeded.
Remove .gitlab-ci.yml make target.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[ann.morin.1998@free.fr: manual: no longer needed to update at all]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The rule to create the staging symlink has it depend on BASE_DIR, and
the symlink is created in BASE_DIR, which means that when the symlink
is created, BASE_DIR is updated, and thus made more recent than the
symlink itself.
As a consequence, every time one runs 'make', the symlink will be older
than BASE_DIR, and so will be re-created.
Ditto for the host symlink when the user has elected to have an
out-of-tree host dir.
Fix that by changing to using an order-only dependency.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we delete /usr/share/bash-completion when bash is not enabled.
We need to delete /etc/bash_completion.d too. For example, the jo package
installs files there:
/etc/bash_completion.d/jo.bash
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some toolchains, like the Linaro gcc7 toolchains, now install libstdc++ debug
library symbols to /lib/debug, which can be as large as the library itself.
This commit removes the extra debug content if debugging is not enabled.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we prepare the release, we generate the manual in various formats,
so that it can be consulted locally without needing the miriads of tools
needed to generate it.
However, this creates the temporary .br2-external.* files in the output
directory, and those end up in the release tarball.
This is not a problem in practice, but is not clean.
Run 'distclean' in the output directory, to get rid of everything but
the generated documentation.
Reported-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Post-build scripts may want to do something based on the list of files
installed by a package. However, since commit
0e2be4db8a the final packages-file-lists.txt
file is only created _after_ the post-build scripts.
Move the assembly of the file lists upwards, before the post-build scripts.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In very limited configurations, it is possible to have a case where no
.files-list-staging.txt files are created. In this case:
cat $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) > \
$(BUILD_DIR)/packages-file-list-staging.txt
becomes:
cat > \
$(BUILD_DIR)/packages-file-list-staging.txt
which of course makes the build hang.. forever.
So we fix this by checking the list is not empty. To keep the code
readable, we introduce an intermediate variable to store the list of
these files.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: always create the file, even if empty]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
With BR2_PER_PACKAGE_DIRECTORIES=y, we have the following code in the
main Makefile:
target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
@$(call MESSAGE,"Finalizing target directory")
$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
The per-package-rsync call creates the global $(TARGET_DIR) from the
per-package $(TARGET_DIR). Then, we call the TARGET_FINALIZE_HOOKS.
One of the TARGET_FINALIZE_HOOKS, PURGE_LOCALES, remove locales that
are not desired by the user. It does so using a loop with the
$(wildcard ...) function.
However, the $(wildcard ...) function is expanded at the moment the
rule is evaluated. And with per-package directory, at the time the
rule is evaluated, the global $(TARGET_DIR) is empty, so $(wildcard
...) will return nothing. It is indeed only after the call to
per-package-rsync that the TARGET_DIR will be populated.
This commit fixes that by moving away from $(wildcard ...) and use a
shell test instead, since we are anyway in big block of shell code.
With this, locales are properly purged again when
BR2_PER_PACKAGE_DIRECTORIES=y.
Fixes: c4e6d5c8be ("core: implement per-package SDK and target")
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- make the style look like the code around (no space in front of ;)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This provides generic functions for Qt5 qmake based packages. It will
make it possible to remove lots of redefinition of
QT5_xxx_{CONFIGURE|BUILD|INSTALL_STAGING}_CMDS. Additionally it
provides a generic target install method which will make most of the
package specific commands obsolete.
This is done by re-running the install step of the qmake generated
Makefile with the package build directory prepended (to the
staging/host path). Even though this does create lengthy pathes it
allows for easy separation of the staging files from the host destined
files by just omitting the resulting BUILD_DIR+HOST_DIR path from the
following rsync call to the real target folder. The cleanup of many
files we dont want in target is deferred to the target-finalize
step. In addition to what's being removed already, we also have to
cleanup some Qt5 specific files (prl) and the documentation directory.
This approach was chosen over copying all files recorded in the pkg-files-list
after some discussion which Thomas Petazzoni summed up:
"We don't yet use pkg-files-list really as part of the build
process anywhere, I feel a bit more comfortable at this point with what
Andreas is proposing."
Thanks to this infrastructure, it will be possible to get rid of the
many conditional install commands because qmake already takes care of
this when generating the Makefile install targets with the given or
autodetected configure options of each package.
However, custom install steps may have to remain in cases where a
particular Buildroot option has no corresponding setting in the
packages configuration options.
Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Several users of rolling-release distributions have been reporting on
IRC that Buildroot is broken now that they have switched to the newly
released make 4.3.
It turns out that the constructs we use to generated and include the
internal br2-external related fragments is no longer working with
make-4.3.
Indeed, an upstream bug report [0] seems to imply that it so far was
working by chance. There has been no further feedback, whether this is
really considered a fix for a previous ill-defined behaviour, or an
actual regression...
In the meantime, we add a workaround, suggested in that same bug report,
that fixes the issue for make 4.3, and that should not break on older
make versions either (verified on all relevant versions: from 3.81,
3.82, 4.0, 4.1, and 4.2).
[0] https://savannah.gnu.org/bugs/?57676
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Mircea Gliga <mgliga@bitdefender.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The comment "Check files that are touched by more than one package"
was previously located right before the calls to the check-uniq-files
script. However, this script and the logic calling it have been
removed in commit 2496189a42 ("core:
drop check-uniq-files"), so the comment no longer makes any sense:
let's drop it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.
This works well when packages are built sequentially, but badly fails
when using top-level parallel build.
More specifically, top-level parallel build can fail with:
comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory
Because that file has been removed concurrently by the build process
of another package.
This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.
At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.
There are however some differences:
(1) The files are no longer ordered in build order, but by alphabetic
ordering of packages. Indeed, "build order" no longer makes any
sense in the context of top-level parallel build.
(2) Some files which were incorrectly tracked are no longer
tracked. For example, the toolchain package is a target package,
but it installs files in $(HOST_DIR). In the previous logic, the
files installed by the toolchain package in $(HOST_DIR) were
incorrectly affected to the next host package that was installed
after the toolchain package. With our new logic, those files are
no longer tracked at all. To fix this, we would have to change
the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
installation steps, not just for the install-host, install-target
and install-staging steps respecitively. But the result was
already incorrect anyway, and therefore this should be fixed
separately.
Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).
Fixes:
http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 509db3b88a added calls to (parts of) the instrumentation steps.
However, those calls are echoed, unlike the other places where we call
them (in the package infra).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Acked-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Create the staging symlink the same way as the host symlink. This means
using a make dependency rather than recreating it every time.
In coreutils versions below 8.27, re-creation of symbolic links was not
atomic. This means that there is a period in time where the existing link is
removed, before the new one is created. In coreutils 8.27 this was fixed,
see [1]. Note that CentOS 7 ships with coreutils 8.22.
In the following scenario, this is a problem:
- an application is compiled using the sysroot prepared by Buildroot and
links against Xenomai userspace libraries, but its build process is steered
from outside of Buildroot
- to know the correct flags, the application makefile uses the 'xeno-config'
file to request them, and passes DESTDIR=/buildroot/output/staging
- the xeno-config responds with flags based on the path
'/buildroot/output/staging/...'
- while the application build is ongoing, a 'make' happens in Buildroot,
causing the 'staging' symlink to be recreated (even though it already
existed)
- when exactly at this time, the application calls the compiler with -I
flags pointing to output/staging, the build fails with:
-I/buildroot/output/staging/usr/include/xenomai/mercury: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai/xenomai: Error: ^ is not a directory
-I/buildroot/output/staging/usr/include/xenomai/psos: Error: ^ is not a directory
Failed: ** ^ *
Work around this problem by only creating the staging symlink once, similar
to how the host symlink (if any) is created.
See also commit d0f4f95e39 which changed the
way these symlinks are made. The reasoning in this commit is to move away
from the 'dirs' target.
[1] 376967889e
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The package instrumentation step 'step_pkg_size' is populating the files:
output/build/packages-file-list.txt
output/build/packages-file-list-staging.txt
output/build/packages-file-list-host.txt
by comparing the list of files before and after installation of a package,
with some clever tricks to detect changes to existing files etc.
As an optimization, instead of gathering this list before and after each
package, where the 'after-state' of one package is the same as the
'before-state' of the next package, only the 'after-state' is used and
is shared between packages.
This works fine, except at the end of the build, as explained next.
In the target-finalize step, many files will be touched. For example, files
like /etc/hosts, /etc/os-release, but also all object files that are
stripped, and all files touched by post-build scripts or created by rootfs
overlays. This means that the 'after-state' of the last package does not
reflect the actual situation after target-finalize is run.
For a single complete build this poses no problem. But, if one incrementally
rebuilds a package after the initial build, e.g. with 'make foo-rebuild',
then all changes that happened in target-finalize at the end of the initial
build (the 'after-state' of the last package built) will be detected as
changes caused by the rebuild of package foo. As a result, all these files
will incorrectly be treated as 'owned' by package foo.
Correct this situation by capturing a new state at the end of
target-finalize, so that the 'before-state' of an incremental build will be
correct.
Note: the reasoning above talks about packages-file-list.txt and
target-finalize, but also applies to
packages-file-list-staging.txt/staging-finalize and
packages-file-list-host.txt/host-finalize.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With per-package folder support, top-level parallel build becomes
safe, so we can enclose the .NOTPARALLEL statement in a
!BR2_PER_PACKAGE_DIRECTORIES condition.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
