When one is applying patches, it is pretty common to end up with .orig
and/or .rej files lying around. Unfortunately, our 'Config.*' match in
check-package ends up matching those files, causing false positives
when running "make check-package". To avoid this, this commit
excludes *.orig and *.rej files for the find logic used in the
check-package target.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
printvars returns nothing when VARS is not passed or empty. This is done
on purpose, see commit fd5bd12379 ("Makefile: printvars: don't print
anything when VARS is not set").
An error message making explicit what is required from the user in order
to use printvars is however better than silently doing nothing.
This adds a check for a non-empty VARS variable.
Cc: Quentin Schulz <foss+buildroot@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 5c54c3ef3d (Makefile: workaround make 4.3 issue for 'printvars
and 'show-vars') did not fully fix the show-vars case, which still
segfaults.
Overall, show-vars generates a JSON blurb. That is supposed to be
machine-readable, so we do not care that the variables are sorted, so
we get rid of it to (slightly) simplify the code.
Then, we currently iterate twice on the list of variables: the first one
to filter-out the 'internal' variables, and the second one to filter
only the variables matching the pattern. We can do away by iterating
only once, and applying both filters at once.
Since we now have an 'and' condition, we can take advantage of it: when
none of the items in $(and) are empty, $(and) evaluates to the last
item, while it evaluates to empty if any of the items is empty. So we
can coalesce the $(if) and $(and) together: $(if $(and a,b),c) is
equivalent to: $(and a,b,c) ; this gains us one parentheses depth.
Finally, the cause for the segfault is an overly-long call to $(info).
Reducing that is not easy: we want to call clean-json on the whole of
the JSON blurb, so we can't emit the individual variables one by one, or
the trailing comma would not be trimmed away.
So, we go crazy: we just output each word from clean-json with $(info).
We can do that, because mk-json-str transforms all spaces in a string
to an escaped UTF-8 sequence, so we will never have spaces in values;
the keys are the variables, so they won't have spaces either; spaces in
the rest of the JSON blurb are totally optional, so we don't care how
many there are. We know there are spaces, because we explicitly
introduce some (after "expanded" or "raw", for example), so we should
never hit a too-big word for $(info) to print.
Thanks to Henri for the suggestion to push $(info) further inside the
macro.
Reported-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Roosen Henri <Henri.Roosen@ginzinger.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Event though the bug with make 4.3 has been reported and fixed, there
has not been a release of make with the fix for a long time, see [1].
As the root cause seems the 'filter' command cannot handle large
chunks of data, like .VARIABLES, we can workaround the problem by
using a foreach command over .VARIABLES, then use the filter command.
It might not be logical to program it that way, but at least the
functionality is now usable.
[1] https://savannah.gnu.org/bugs/?59093#comment10
Signed-off-by: Henri Roosen <henri.roosen@ginzinger.com>
[yann.morin.1998@free.fr: add comment to reference the bug]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Back many years ago, we developed an Eclipse plugin that simplified
the usage of Buildroot toolchains. Enabling the BR2_ECLIPSE_REGISTER=y
was registering the Buildroot toolchain into a special file in your
HOME folder that the Eclipse plugin would recognize to allow to
directly use the Buildroot cross-compiler.
This Eclipse plugin has not been maintained for years. The last commit
in the repository dates back from September 2017. Since then Eclipse
has moved on, and the plugin is no longer compatible with current
versions of Eclipse.
Also, Eclipse is probably no longer that widely used in the embedded
Linux space, as other more modern IDEs have become more popular.
All in all, it's time to say good bye to this Eclipse integration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Now that all hash files have been fixed, enable checking of hash
spacing in check-package.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In eudev and systemd, we have code that deletes the hwdb sources from
the target - they are not useful since a binary hwdb is created from
them. However, if eudev or systemd is not used, then those sources are
not useful either. It's possible that other packages than eudev or
systemd install hwdb files, which would be left on the system.
Always remove the hwdb files.
Note that we don't expect much space savings from this, but anything may
help. It's certainly more consistent to do it always than just in eudev
and systemd.
We do this both from /usr/lib/udev (usual installation path for systemd)
and in /etc/udev (usual installation path for eudev) because packages
may install in either location.
We keep the comment explaining why it's done in rootfs-pre-cmd instead
of target-finalize - this was only present in eudev.mk.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
this directory is used by the rpm package manager, and packages
like systemd will install "macros" for this system.
It should be deleted just like the similar
/usr/share/aclocal directory from Autoconf.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Until now, when BR2_CCACHE=y, ccache support was built into the
toolchain wrapper, and used regardless of whether the toolchain is
using during the Buildroot build itself, or later as part of the SDK.
However, having ccache support forcefully enabled in the SDK can
really be surprising, and is certainly unexpected for a
cross-compilation toolchain. This can be particularly surprising as
the ccache cache directory may be hardcoded in the ccache binary to
point to a folder that does not make sense on the SDK user's machine.
So what this commit does is create a BR2_USE_CCACHE variable, which
when set to 1 tells the toolchain wrapper to use ccache. Not defining
the variable, or specifying any other value that 1 causes the
toolchain wrapper to not use ccache. The main Buildroot Makefile is
modified to export BR2_USE_CCACHE = 1 when ccache support is enabled,
so that ccache is used during the Buildroot build.
However, when someone will use the SDK outside of Buildroot, the
toolchain wrapper will not use ccache.
The BR2_USE_CCACHE variable is only conditionally enabled in the main
Makefile (via ?=) so that it can be overridden in the environment if
one wants to quickly test disabling ccache in a ccache-enabled
Buildroot configuration. This is the scenario that was considered in
commit 792f1278e3 ("toolchain-wrapper:
support change of BR2_CCACHE"), which added the BR_NO_CCACHE variable.
The BR_NO_CCACHE variable is no longer needed, and replaced by this
BR2_USE_CCACHE variable.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
[Thomas: almost entirely rework the implementation and commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds support for a new type of graph, showing the timeline
of a build. It shows, with one line per package, when each of this
package steps started/ended, and therefore allows to see the
sequencing of the package builds.
For a fully serialized build like we have today, this is not super
useful (except to show that everything is serialized), but it becomes
much more useful in the context of top-level parallel build.
We chose to order the graph by the time-of-configure, as it is the
closest to the actual cascade-style of a true dependency graph, which is
tiny bit more complex to achieve properly. The actual result still looks
pretty good.
The graph-build make target is extended to also generate this new
timeline graph.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- sort by start-of-configure time
- re-use existing colorsets (default or alternate)
- fix python2isms
- fix check-package
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Virtual packages are not added to the _RDEPENDENCIES list of packages
that they depend on (i.e. their provider).
This causes <provider>-show-rdepends to not show the virtual package
and <provider>-show-recursive-rdepends to miss all the packages that
transitively depend on <provider> via the virtual package.
The virtual make targets (e.g. <pkg>-show-info) are also not marked as
phony for virtual packages.
To fix those issues, remove most of the special handling of virtual
packages in pkg-generic by making $($($(1)_KCONFIG_VAR))=y for them as
well.
This also allows removal of some duplicated code in pkg-generic.mk and a
now unneeded special condition in CHECK_ONE_DEPENDENCY.
Still keep the virtual package out of PACKAGES since there is e.g. no
need to rsync per-package target dir to global target dir. I am not
aware of any showstoppers preventing addition to PACKAGES as well,
though, so it is probably just an optimization.
Signed-off-by: Anssi Hannula <anssi.hannula@bitwise.fi>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
U-Boot looks for the environment variable DEVICE_TREE and uses its value if
set instead of the CONFIG_DEFAULT_DEVICE_TREE configuration option since
v2021.01, more specifically commit c0f1ebe9c1b9745e (binman: Allow selecting
default FIT configuration) - So unexport it like we do for other
"troublesome" environment variables to ensure consistent behaviour.
Reported-by: Neal Frager <nealf@xilinx.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The variable 'KERNEL_ARCH' is actually a normalized version of
'ARCH'/'BR2_ARCH'. For example, 'arcle' and 'arceb' both become 'arc', just
as all powerpc variants become 'powerpc'.
It is presumably called 'KERNEL_ARCH' because the Linux kernel is typically
the first place where support for a new architecture is added, and thus is
the entity that defines the normalized name.
However, the term 'KERNEL_ARCH' can also be interpreted as 'the architecture
used by the kernel', which need not be exactly the same as 'the normalized
name for a certain arch'. In particular, for cases where a 64-bit
architecture is running a 64-bit kernel but 32-bit userspace. Examples
include:
* aarch64 architecture, with aarch64 kernel and 32-bit (ARM) userspace
* x86_64 architecture, with x86_64 kernel and 32-bit (i386) userspace
In such cases, the 'architecture used by the kernel' needs to refer to the
64-bit name (aarch64, x86_64), whereas all userspace applications need to
refer the, potentially normalized, 32-bit name.
This means that there need to be two different variables:
KERNEL_ARCH: the architecture used by the kernel
NORMALIZED_ARCH: the normalized name for the current userspace architecture
At this moment, both will actually have the same content. But a subsequent
patch will add basic support for situations described above, in which
KERNEL_ARCH may become overwritten to the 64-bit architecture, while
NORMALIZED_ARCH needs to remain the same (32-bit) case.
This commit replaces use of KERNEL_ARCH where actually the userspace arch is
needed. Places that use KERNEL_ARCH in combination with building of kernel
modules are not touched.
There may be cases where a package builds both a kernel module as userspace,
in which case it may need to know about both KERNEL_ARCH and
NORMALIZED_ARCH, for the case where they differ. But this is to be fixed on
a per-need basis.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
[Arnout: Also rename BR2_KERNEL_ARCH to BR2_NORMALIZED_ARCH]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Similar to other arch-specific strings, the 'KERNEL_ARCH' variable can be
determined from Config.in.<arch> files.
Besides aligning with similar strings, this also means simplification: the
big 'sed' covers several architectures not even supported by Buildroot.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The seperation of the fields in the hash file should be 2 spaces for
consistency.
Since a large number of hash files still violate this rule, exclude it
from "make check-package" (and thus from CI).
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout:
- Move it to a separate class, so it can be excluded.
- Exclude it from "make check-package"
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
We introduce the concept of a pre-build script that works similar to
the already existing post-build and post-image scripts.
The pre-build script(s) are executed before the build commences. This
allows a user to run some preperatory tasks prior to the build.
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Make 4.3 is buggy and leads to a "Segmentation fault (core dumped)" when
calling 'make printvars' or 'make show-vars', so let's refuse to execute
those recipes if Make 4.3 by adding 'check-make-version' recipe as
depedendency of 'printvars' and 'show-vars' as suggested by Yann E. Morin.
Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The current printvars output suffers from a serious design flaw:
variables are not delimited, which makes it impossible to reliably
retrieve the value of variables; only variables that are known to
not contain a \n can be relatively safely extracted.
However, in some cases, it is important to be able to retrieve the
multi-line value of a variable, notably the CMDS or the hooks. One
such use-case (to follow in an unscheduled future) would be to hash
the variables that make up a package "configuration", and cache or
extract the files for that package to speed up the build.
Modeled after printvars and show-info, we introduce show-vars (what a
lack of imagination here) that outputs a json dictionary which keys are
the variable names, and for each variable, provides the raw and expanded
values.
Unlike printvars, we do not provide a way to get either the raw or
expanded value; both are systematically printed; a user will get just
the one is needs. Additionally, strings in JSON are quoted, so there is
no need to provide a way to quote variables; that would not make sense.
Note: for printvars, we require that the user provides an explicit
pattern to filter variables on. This is historical (see fd5bd12379,
Makefile: printvars: don't print anything when VARS is not set). The
underlying reasoning was that printvars is too "raw", and variables are
not well delimited, so printvars was mostly used to extract a few values
here and there, from scripts, or to quickly inspect a specific package's
variables during debugging.
But show-vars, although technically plain-text, being JSON, is not very
human-readable, and is mostly aimed at tools that will parse it with a
real JSON parser, and which will want to have a complete view of a lot
of variables at once. As such, and contrary to printvars, it makes sense
to report on all variables by default, unless the user explicitly
requested a subset.
As a final note: a lot of our variables only make sense in the context
of an actual make target. For example, a variable of package foo, that
contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is
because our CMDS and hooks are expanded as the recipe of a stamp file
that lies in the package build directory.
But for show-info, this falls flat on its face: it is not the stamp file
of a package, so there is no package directory, and show-info itself has
not directory part, so $(@D) expands to '.' (dot).
Additionally, some variables may contain calls to $(shell) (e.g. to call
pkg-config), and this also does not work with show-info.
These two issues make it impossible to emit the correct expanded value
of variables. To be noted: printvars has the exact same limitations for
the exact same reasons.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Set HOSTCC_NOCCACHE and HOSTCXX_NOCCACHE only if they are not
set. This allows recursive calls to "make" to work as intended in the
presence of ccache. Such recursive calls to "make" can for example
happen if one calls "make legal-info" from within a post-build script,
to integrate some results of the legal-info output into the root
filesystem.
Without guarding these variables, a recursive invocation of make would
re-define
HOSTCC_NOCCACHE := $(HOSTCC)
and
HOSTCXX_NOCCACHE := $(HOSTCXX)
at a point in time when HOSTCC and HOSTCXX already point to ccache.
It used to work by "accident" until
ca6a2907c2 ("make: support: use `command
-v' instead of `which'"), due to how "which" was behaving when invoked
with multiple arguments. After switching to "command -v", which
behaves different with multiple arguments, this HOSTCC_NOCCACHE
redefinition problem surfaced. Even though
ca6a2907c2 has since then been reverted
for other reasons, it does make sense to guard the definition of
HOSTCC_NOCCACHE and HOSTCXX_NOCCACHE to not rely on a side-effect of
using "which".
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since a long time, we have a check-package check for patches. Make sure
that this check runs in 'make check-package', by including *.patch in
the find expression.
There are still a number of patches without SoB, and these are not so
trivial to fix, so for now, disable the SoB check.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Ricardo: do not run check for SoB for now]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
The unmatched escaped single-quote lies in the middle of a few
function calls, so they too must be fake-closed to properly fix
colour highlighting in some editors.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
xz compresses better than bzip2, and is (getting) more popular, so build
release tarballs as .tar.xz (in addition to .tar.gz) instead of .tar.bz2,
similar to how the kernel did ~8 years ago:
https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html
-rw-r--r-- 1 peko peko 5,1M Dec 2 17:55 buildroot-2021.11-rc3.tar.xz
-rw-r--r-- 1 peko peko 5,7M Nov 30 18:15 buildroot-2021.11-rc3.tar.bz2
-rw-r--r-- 1 peko peko 6,8M Nov 30 18:15 buildroot-2021.11-rc3.tar.gz
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
`which' has been discontinued after 2.21 release in 2015 due this (git
repository is empty [1]) and version shipped in Debian produces warning
[2]:
/usr/bin/which: this version of `which' is deprecated; use `command -v' in scripts instead.
`command is POSIX [3] and supported on all common shells (bash, zsh,
dash, busybox sh, mksh).
Patch tested on dash as the default shell.
[1] https://git.savannah.gnu.org/cgit/which.git
[2] 3a8dd10b45
[3] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Parallelizes locale generation based on `BR2_JLEVEL` setting.
Locale generation always runs during the finalize stage and can consume
a significant amount of time. Parallelizing it greatly reduces that time
on multi-core machines.
To parallelize it, we first invoke `localedef` for every locale in
parallel with the `--no-archive` option. This creates the intermediate
locale data instead of writing to the finally archive directly.
Then, we invoke `localedef` again once to create the archive from the
intermediate compiled locale data files.
We have to do it this way because `localedef` does not do any locking
when writing to the archive file, so calling it without `--no-archive`
concurrently could result in a corrupt archive file or an archive file
that is missing some locales.
While we're at it, make two additional improvements:
- Remove locale-archive before adding to it. Otherwise, repeated
applications of target-finalize will keep on growing the file.
- Sort the locales when creating locale-archive so its contents are
reproducible.
We use `find` to collect the installed locales rather than LOCALES. This
makes it possible for something else (skeleton, overlay, custom package)
to create and install additional locales and still have them added to
locale-archive.
Signed-off-by: Gleb Mazovetskiy <glex.spb@gmail.com>
[Arnout:
- Remove -j$(PARALLEL_JOBS), it's already part of $(MAKE)
- Remove HOST_DIR, TARGET_DIR, STAGING_DIR, they're already exported
- Extend commit message
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Like commit 1f187371d0 for cpe-updates data, also remove pkg-stats
data on clean.
Unlike the rest, those are not nicely located in a directory of their
own, and have no variable name associated with them, so we just need
to repeat their names in the clean rule.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit fd7312940a (Makefile: add new missing-cpe target) added the
rule to generate a set of files to update the NVD.
For an in-tree build, 'make clean' remove the output directory, so
those files are removed. But for an out-of-tree build, the output
directory is not removed, so those files still linger around after a
clean.
Explicitly remove them on clean, to cater for both cases.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Matthew Weber <matthew.weber@collins.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 7d17ae2acf (.flake8: fix check for 80/132 columns) introduced a
difference in how flake8 behaves between the automatic checks done in
the CI, where the maximum line length is 132, and the local checks,
where the maximum line length is 80.
The rationale at the time was that we recommend 80 char lines, but that
we accept 132 when it makes sense for readability.
However, this is very annoying when running flake8 locally, because of
two reasons:
1. human reviews on python scripts have not been as thorough as we did
expect; indeed, we've let a lot of long lines slip through; this
causes a lot of spurious failures that hide away the actual errors;
2. when hacking on a python script, the issues reported will not be
caused by the current changes, so the many reported failures
actually hide away the newly introduced issues.
Additionally, our 'make check-flake8' rule already enforces the 132-char
limit, and the issues reported are different than when manually running
flake8 on individual files.
Furthermore, the readability rationale for the 80-char limit is
definitely shattered by the mere rationale of allowing 132-char limit
for... readability...
We've arrived to a point where this separation is causing our checks
around flake8 to become mostly unusable and useless, as they do not
report meaningful issues, and people are no longer paying attention, and
this has caused actual issues to be introduced.
Finally, terminal emulators of today have long lifted the 80-char limit,
and are more than capable of displaying 132-char wide lines.
Switch back to using a 132-char limit.
This reverts commit 7d17ae2acf.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Now that pkg-stats is not just a maintainer-oriented tool, but a tool
generally useful to users, introduce a make target to run
pkg-stats. Of course, it is run with the newly introduced -c option,
which produces a pkg-stats output for just the selection of packages
of the currently defined configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Although BR2_DL_DIR is indeed a site-local setting, which does not
actually define the target system, we've had it in the tree for a
long time now, and people have been depending on it for a variety
of use-cases.
Furthermore, BR2_DL_DIR is far from the only such site-local setting,
BR2_CCACHE_DIR springs to mind, and in the less-obvious category, we
can also find BR2_JLEVEL, but also BR2_WGET, BR2_SVN, BR2_GIT et al.
as they may be tweaked to set the timeout, number of retries or so on
to work around stupid proxies. But of course, the most local site-local
setting is probably BR2_PACKAGE_OVERRIDE_FILE, with its default value
being explicitly just 'local.mk'.
Ideally, we would like to have a clear separation between the
configuration that actually defines the target system on one hand,
and the site-local settings that drive and control how the build is
performed, on the other hand. This is by far a much bigger endeavour
than just dropping BR2_DL_DIR from the saved defconfig.
This reverts commit 36edacce9c (adapted
to keep the fix from 1a7873ec98).
Closes: #13291
Note: thanks to Thomas; some phrasing above was borrowed from a
discussion with him.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Lance Fredrickson <lancethepants@gmail.com>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
When we savedefconfig, we remove BR2_DEFCONFIG (f71a621d91, savedefconfig:
Remove BR2_DEFCONFIG from saved defconfig file) and BR2_DL_DIR (36edacce9c,
Makefile: exclude BR2_DL_DIR from savedefconfig), because their meaning
is only valid locally.
However, we were not careful to really match the exact variables, so we
could match arbitrary options.
For example, these config options would all be dropped:
BR2_DEFCONFIG="toto"
BR2_DL_DIR="titi"
BR2_PACKAGE_SABR2_DEFCONFIG="tutu"
BR2_PACKAGE_SABR2_DL_DIR=y
While the first two are indeed the ones we want to drop, the last two
are options (whatever their meaning or how poorly named they are) of the
hypothetical 'sabr2' package, and we want to keep those in a defconfig.
When cleaning the just-saved defconfig, be sure to anchor the patterns to
the beginning of the line.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Herve Codina <Herve.CODINA@celad.com>
Acked-by: Herve Codina <Herve.CODINA@celad.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running "make savedefconfig" with BR2_DL_DIR set also saves that
variable, which is only useful in local context.
Signed-off-by: Sven Oliver Moll <buildroot@svol.li>
[yann.morin.1998@free.fr: wrap line]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: drop supperfluous depends on s390x in choice]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Move the code to run check-flake8 into the Makefile, like we have for
check-package, so that it is easy to run locally (and not wait for
someone to report a failure from their Gitlab pipelines).
Compared to the existing check from gitlab-ci.yml, the Makefile check
differs in this respect:
- don't explicitly find *.py files: they are supposed to also be found
as a result of running 'file' on them;
- use git ls-tree instead of find: this is supopsedly faster as it
uses the index rather than readdir();
- don't output the count of warnings or errors: the output is a single
integer, which is confusing when there are errors, and even more so
when there are no, when it is simply '0';
- don't sort: the output is already stable and independent from the
locale;
- don't report the number of processed files: this information is
rather useless, and getting a hold of it would be more challenging
in this new code.
Note: ideally, we would want to use --null, --zero, or similar options,
with utilities that generates or parses a files listing. While git
ls-tree and xargs do support it, it becomes a little bit tricky to use
the --print0 option of file, and then grep in that output (it is not
undoable, but would requires replacing grep+cut with some sed trickery).
Since we do not expect our scripts names to contain funky chars (like
\n or a colon), we just hand-wave away that issue (and the old code was
doing the same assumption too).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As 18f6c26118 just did to silence the file lists commands, switch to
using $(Q) instead of a plain @, to silence the commands.
Using $(Q) will allow to debug the commands with V=1.
We keep @ for the calls to MESSAGE, though.
The commands that are not currently silenced are left as-is, and they
can be converted to being silent in a followup patch, if need be,
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since commit 0e2be4db8a
("package/pkg-generic: make file list logic parallel build
compatible"), the commands executed at the every end of the build
to assemble the list of files installed by the different packages
are visible in the make output. They are quite noisy, and clutter
the output.
The other commands in target-finalize are also hidden using "@",
so we should also do the same for those commands. But that hurts
debuggability, so we use $(Q) (the existing '@'s can be changed
in a followup patch).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use '$(Q)', not '@']
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
BR2_VERSION_FULL is currently defined as follows:
BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.
The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.
If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).
This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.
The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.
Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:
$ make print-version
2020.08-git-00758-gc351877a6e
So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:
$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e
This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.
So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.
In the above situation, with this commit applied, we get:
$ make print-version
dummy-tag-6-g6258cdddeb
(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)
Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since Gitlab 12.9, Gitlab allow to trigger child pipeline with generated configuration file.
See: https://gitlab.com/gitlab-org/gitlab/-/issues/35632
This allow us to stop updating the .gitlab-ci.yml file when a
new defconfig is added to Buildroot.
Remove check-gitlab-ci.yml job since it is now uneeded.
Remove .gitlab-ci.yml make target.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[ann.morin.1998@free.fr: manual: no longer needed to update at all]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The rule to create the staging symlink has it depend on BASE_DIR, and
the symlink is created in BASE_DIR, which means that when the symlink
is created, BASE_DIR is updated, and thus made more recent than the
symlink itself.
As a consequence, every time one runs 'make', the symlink will be older
than BASE_DIR, and so will be re-created.
Ditto for the host symlink when the user has elected to have an
out-of-tree host dir.
Fix that by changing to using an order-only dependency.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we delete /usr/share/bash-completion when bash is not enabled.
We need to delete /etc/bash_completion.d too. For example, the jo package
installs files there:
/etc/bash_completion.d/jo.bash
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some toolchains, like the Linaro gcc7 toolchains, now install libstdc++ debug
library symbols to /lib/debug, which can be as large as the library itself.
This commit removes the extra debug content if debugging is not enabled.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we prepare the release, we generate the manual in various formats,
so that it can be consulted locally without needing the miriads of tools
needed to generate it.
However, this creates the temporary .br2-external.* files in the output
directory, and those end up in the release tarball.
This is not a problem in practice, but is not clean.
Run 'distclean' in the output directory, to get rid of everything but
the generated documentation.
Reported-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Post-build scripts may want to do something based on the list of files
installed by a package. However, since commit
0e2be4db8a the final packages-file-lists.txt
file is only created _after_ the post-build scripts.
Move the assembly of the file lists upwards, before the post-build scripts.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In very limited configurations, it is possible to have a case where no
.files-list-staging.txt files are created. In this case:
cat $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) > \
$(BUILD_DIR)/packages-file-list-staging.txt
becomes:
cat > \
$(BUILD_DIR)/packages-file-list-staging.txt
which of course makes the build hang.. forever.
So we fix this by checking the list is not empty. To keep the code
readable, we introduce an intermediate variable to store the list of
these files.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: always create the file, even if empty]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
With BR2_PER_PACKAGE_DIRECTORIES=y, we have the following code in the
main Makefile:
target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
@$(call MESSAGE,"Finalizing target directory")
$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
The per-package-rsync call creates the global $(TARGET_DIR) from the
per-package $(TARGET_DIR). Then, we call the TARGET_FINALIZE_HOOKS.
One of the TARGET_FINALIZE_HOOKS, PURGE_LOCALES, remove locales that
are not desired by the user. It does so using a loop with the
$(wildcard ...) function.
However, the $(wildcard ...) function is expanded at the moment the
rule is evaluated. And with per-package directory, at the time the
rule is evaluated, the global $(TARGET_DIR) is empty, so $(wildcard
...) will return nothing. It is indeed only after the call to
per-package-rsync that the TARGET_DIR will be populated.
This commit fixes that by moving away from $(wildcard ...) and use a
shell test instead, since we are anyway in big block of shell code.
With this, locales are properly purged again when
BR2_PER_PACKAGE_DIRECTORIES=y.
Fixes: c4e6d5c8be ("core: implement per-package SDK and target")
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- make the style look like the code around (no space in front of ;)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This provides generic functions for Qt5 qmake based packages. It will
make it possible to remove lots of redefinition of
QT5_xxx_{CONFIGURE|BUILD|INSTALL_STAGING}_CMDS. Additionally it
provides a generic target install method which will make most of the
package specific commands obsolete.
This is done by re-running the install step of the qmake generated
Makefile with the package build directory prepended (to the
staging/host path). Even though this does create lengthy pathes it
allows for easy separation of the staging files from the host destined
files by just omitting the resulting BUILD_DIR+HOST_DIR path from the
following rsync call to the real target folder. The cleanup of many
files we dont want in target is deferred to the target-finalize
step. In addition to what's being removed already, we also have to
cleanup some Qt5 specific files (prl) and the documentation directory.
This approach was chosen over copying all files recorded in the pkg-files-list
after some discussion which Thomas Petazzoni summed up:
"We don't yet use pkg-files-list really as part of the build
process anywhere, I feel a bit more comfortable at this point with what
Andreas is proposing."
Thanks to this infrastructure, it will be possible to get rid of the
many conditional install commands because qmake already takes care of
this when generating the Makefile install targets with the given or
autodetected configure options of each package.
However, custom install steps may have to remain in cases where a
particular Buildroot option has no corresponding setting in the
packages configuration options.
Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Several users of rolling-release distributions have been reporting on
IRC that Buildroot is broken now that they have switched to the newly
released make 4.3.
It turns out that the constructs we use to generated and include the
internal br2-external related fragments is no longer working with
make-4.3.
Indeed, an upstream bug report [0] seems to imply that it so far was
working by chance. There has been no further feedback, whether this is
really considered a fix for a previous ill-defined behaviour, or an
actual regression...
In the meantime, we add a workaround, suggested in that same bug report,
that fixes the issue for make 4.3, and that should not break on older
make versions either (verified on all relevant versions: from 3.81,
3.82, 4.0, 4.1, and 4.2).
[0] https://savannah.gnu.org/bugs/?57676
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Mircea Gliga <mgliga@bitdefender.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The comment "Check files that are touched by more than one package"
was previously located right before the calls to the check-uniq-files
script. However, this script and the logic calling it have been
removed in commit 2496189a42 ("core:
drop check-uniq-files"), so the comment no longer makes any sense:
let's drop it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.
This works well when packages are built sequentially, but badly fails
when using top-level parallel build.
More specifically, top-level parallel build can fail with:
comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory
Because that file has been removed concurrently by the build process
of another package.
This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.
At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.
There are however some differences:
(1) The files are no longer ordered in build order, but by alphabetic
ordering of packages. Indeed, "build order" no longer makes any
sense in the context of top-level parallel build.
(2) Some files which were incorrectly tracked are no longer
tracked. For example, the toolchain package is a target package,
but it installs files in $(HOST_DIR). In the previous logic, the
files installed by the toolchain package in $(HOST_DIR) were
incorrectly affected to the next host package that was installed
after the toolchain package. With our new logic, those files are
no longer tracked at all. To fix this, we would have to change
the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
installation steps, not just for the install-host, install-target
and install-staging steps respecitively. But the result was
already incorrect anyway, and therefore this should be fixed
separately.
Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).
Fixes:
http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
