We introduce the concept of a pre-build script that works similar to
the already existing post-build and post-image scripts.
The pre-build script(s) are executed before the build commences. This
allows a user to run some preperatory tasks prior to the build.
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Make 4.3 is buggy and leads to a "Segmentation fault (core dumped)" when
calling 'make printvars' or 'make show-vars', so let's refuse to execute
those recipes if Make 4.3 by adding 'check-make-version' recipe as
depedendency of 'printvars' and 'show-vars' as suggested by Yann E. Morin.
Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The current printvars output suffers from a serious design flaw:
variables are not delimited, which makes it impossible to reliably
retrieve the value of variables; only variables that are known to
not contain a \n can be relatively safely extracted.
However, in some cases, it is important to be able to retrieve the
multi-line value of a variable, notably the CMDS or the hooks. One
such use-case (to follow in an unscheduled future) would be to hash
the variables that make up a package "configuration", and cache or
extract the files for that package to speed up the build.
Modeled after printvars and show-info, we introduce show-vars (what a
lack of imagination here) that outputs a json dictionary which keys are
the variable names, and for each variable, provides the raw and expanded
values.
Unlike printvars, we do not provide a way to get either the raw or
expanded value; both are systematically printed; a user will get just
the one is needs. Additionally, strings in JSON are quoted, so there is
no need to provide a way to quote variables; that would not make sense.
Note: for printvars, we require that the user provides an explicit
pattern to filter variables on. This is historical (see fd5bd12379,
Makefile: printvars: don't print anything when VARS is not set). The
underlying reasoning was that printvars is too "raw", and variables are
not well delimited, so printvars was mostly used to extract a few values
here and there, from scripts, or to quickly inspect a specific package's
variables during debugging.
But show-vars, although technically plain-text, being JSON, is not very
human-readable, and is mostly aimed at tools that will parse it with a
real JSON parser, and which will want to have a complete view of a lot
of variables at once. As such, and contrary to printvars, it makes sense
to report on all variables by default, unless the user explicitly
requested a subset.
As a final note: a lot of our variables only make sense in the context
of an actual make target. For example, a variable of package foo, that
contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is
because our CMDS and hooks are expanded as the recipe of a stamp file
that lies in the package build directory.
But for show-info, this falls flat on its face: it is not the stamp file
of a package, so there is no package directory, and show-info itself has
not directory part, so $(@D) expands to '.' (dot).
Additionally, some variables may contain calls to $(shell) (e.g. to call
pkg-config), and this also does not work with show-info.
These two issues make it impossible to emit the correct expanded value
of variables. To be noted: printvars has the exact same limitations for
the exact same reasons.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Set HOSTCC_NOCCACHE and HOSTCXX_NOCCACHE only if they are not
set. This allows recursive calls to "make" to work as intended in the
presence of ccache. Such recursive calls to "make" can for example
happen if one calls "make legal-info" from within a post-build script,
to integrate some results of the legal-info output into the root
filesystem.
Without guarding these variables, a recursive invocation of make would
re-define
HOSTCC_NOCCACHE := $(HOSTCC)
and
HOSTCXX_NOCCACHE := $(HOSTCXX)
at a point in time when HOSTCC and HOSTCXX already point to ccache.
It used to work by "accident" until
ca6a2907c2 ("make: support: use `command
-v' instead of `which'"), due to how "which" was behaving when invoked
with multiple arguments. After switching to "command -v", which
behaves different with multiple arguments, this HOSTCC_NOCCACHE
redefinition problem surfaced. Even though
ca6a2907c2 has since then been reverted
for other reasons, it does make sense to guard the definition of
HOSTCC_NOCCACHE and HOSTCXX_NOCCACHE to not rely on a side-effect of
using "which".
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since a long time, we have a check-package check for patches. Make sure
that this check runs in 'make check-package', by including *.patch in
the find expression.
There are still a number of patches without SoB, and these are not so
trivial to fix, so for now, disable the SoB check.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Ricardo: do not run check for SoB for now]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
The unmatched escaped single-quote lies in the middle of a few
function calls, so they too must be fake-closed to properly fix
colour highlighting in some editors.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
xz compresses better than bzip2, and is (getting) more popular, so build
release tarballs as .tar.xz (in addition to .tar.gz) instead of .tar.bz2,
similar to how the kernel did ~8 years ago:
https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html
-rw-r--r-- 1 peko peko 5,1M Dec 2 17:55 buildroot-2021.11-rc3.tar.xz
-rw-r--r-- 1 peko peko 5,7M Nov 30 18:15 buildroot-2021.11-rc3.tar.bz2
-rw-r--r-- 1 peko peko 6,8M Nov 30 18:15 buildroot-2021.11-rc3.tar.gz
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
`which' has been discontinued after 2.21 release in 2015 due this (git
repository is empty [1]) and version shipped in Debian produces warning
[2]:
/usr/bin/which: this version of `which' is deprecated; use `command -v' in scripts instead.
`command is POSIX [3] and supported on all common shells (bash, zsh,
dash, busybox sh, mksh).
Patch tested on dash as the default shell.
[1] https://git.savannah.gnu.org/cgit/which.git
[2] 3a8dd10b45
[3] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Parallelizes locale generation based on `BR2_JLEVEL` setting.
Locale generation always runs during the finalize stage and can consume
a significant amount of time. Parallelizing it greatly reduces that time
on multi-core machines.
To parallelize it, we first invoke `localedef` for every locale in
parallel with the `--no-archive` option. This creates the intermediate
locale data instead of writing to the finally archive directly.
Then, we invoke `localedef` again once to create the archive from the
intermediate compiled locale data files.
We have to do it this way because `localedef` does not do any locking
when writing to the archive file, so calling it without `--no-archive`
concurrently could result in a corrupt archive file or an archive file
that is missing some locales.
While we're at it, make two additional improvements:
- Remove locale-archive before adding to it. Otherwise, repeated
applications of target-finalize will keep on growing the file.
- Sort the locales when creating locale-archive so its contents are
reproducible.
We use `find` to collect the installed locales rather than LOCALES. This
makes it possible for something else (skeleton, overlay, custom package)
to create and install additional locales and still have them added to
locale-archive.
Signed-off-by: Gleb Mazovetskiy <glex.spb@gmail.com>
[Arnout:
- Remove -j$(PARALLEL_JOBS), it's already part of $(MAKE)
- Remove HOST_DIR, TARGET_DIR, STAGING_DIR, they're already exported
- Extend commit message
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Like commit 1f187371d0 for cpe-updates data, also remove pkg-stats
data on clean.
Unlike the rest, those are not nicely located in a directory of their
own, and have no variable name associated with them, so we just need
to repeat their names in the clean rule.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit fd7312940a (Makefile: add new missing-cpe target) added the
rule to generate a set of files to update the NVD.
For an in-tree build, 'make clean' remove the output directory, so
those files are removed. But for an out-of-tree build, the output
directory is not removed, so those files still linger around after a
clean.
Explicitly remove them on clean, to cater for both cases.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Matthew Weber <matthew.weber@collins.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 7d17ae2acf (.flake8: fix check for 80/132 columns) introduced a
difference in how flake8 behaves between the automatic checks done in
the CI, where the maximum line length is 132, and the local checks,
where the maximum line length is 80.
The rationale at the time was that we recommend 80 char lines, but that
we accept 132 when it makes sense for readability.
However, this is very annoying when running flake8 locally, because of
two reasons:
1. human reviews on python scripts have not been as thorough as we did
expect; indeed, we've let a lot of long lines slip through; this
causes a lot of spurious failures that hide away the actual errors;
2. when hacking on a python script, the issues reported will not be
caused by the current changes, so the many reported failures
actually hide away the newly introduced issues.
Additionally, our 'make check-flake8' rule already enforces the 132-char
limit, and the issues reported are different than when manually running
flake8 on individual files.
Furthermore, the readability rationale for the 80-char limit is
definitely shattered by the mere rationale of allowing 132-char limit
for... readability...
We've arrived to a point where this separation is causing our checks
around flake8 to become mostly unusable and useless, as they do not
report meaningful issues, and people are no longer paying attention, and
this has caused actual issues to be introduced.
Finally, terminal emulators of today have long lifted the 80-char limit,
and are more than capable of displaying 132-char wide lines.
Switch back to using a 132-char limit.
This reverts commit 7d17ae2acf.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Now that pkg-stats is not just a maintainer-oriented tool, but a tool
generally useful to users, introduce a make target to run
pkg-stats. Of course, it is run with the newly introduced -c option,
which produces a pkg-stats output for just the selection of packages
of the currently defined configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Although BR2_DL_DIR is indeed a site-local setting, which does not
actually define the target system, we've had it in the tree for a
long time now, and people have been depending on it for a variety
of use-cases.
Furthermore, BR2_DL_DIR is far from the only such site-local setting,
BR2_CCACHE_DIR springs to mind, and in the less-obvious category, we
can also find BR2_JLEVEL, but also BR2_WGET, BR2_SVN, BR2_GIT et al.
as they may be tweaked to set the timeout, number of retries or so on
to work around stupid proxies. But of course, the most local site-local
setting is probably BR2_PACKAGE_OVERRIDE_FILE, with its default value
being explicitly just 'local.mk'.
Ideally, we would like to have a clear separation between the
configuration that actually defines the target system on one hand,
and the site-local settings that drive and control how the build is
performed, on the other hand. This is by far a much bigger endeavour
than just dropping BR2_DL_DIR from the saved defconfig.
This reverts commit 36edacce9c (adapted
to keep the fix from 1a7873ec98).
Closes: #13291
Note: thanks to Thomas; some phrasing above was borrowed from a
discussion with him.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Lance Fredrickson <lancethepants@gmail.com>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
When we savedefconfig, we remove BR2_DEFCONFIG (f71a621d91, savedefconfig:
Remove BR2_DEFCONFIG from saved defconfig file) and BR2_DL_DIR (36edacce9c,
Makefile: exclude BR2_DL_DIR from savedefconfig), because their meaning
is only valid locally.
However, we were not careful to really match the exact variables, so we
could match arbitrary options.
For example, these config options would all be dropped:
BR2_DEFCONFIG="toto"
BR2_DL_DIR="titi"
BR2_PACKAGE_SABR2_DEFCONFIG="tutu"
BR2_PACKAGE_SABR2_DL_DIR=y
While the first two are indeed the ones we want to drop, the last two
are options (whatever their meaning or how poorly named they are) of the
hypothetical 'sabr2' package, and we want to keep those in a defconfig.
When cleaning the just-saved defconfig, be sure to anchor the patterns to
the beginning of the line.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Sven Oliver Moll <buildroot@svol.li>
Cc: Herve Codina <Herve.CODINA@celad.com>
Acked-by: Herve Codina <Herve.CODINA@celad.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running "make savedefconfig" with BR2_DL_DIR set also saves that
variable, which is only useful in local context.
Signed-off-by: Sven Oliver Moll <buildroot@svol.li>
[yann.morin.1998@free.fr: wrap line]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: drop supperfluous depends on s390x in choice]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Move the code to run check-flake8 into the Makefile, like we have for
check-package, so that it is easy to run locally (and not wait for
someone to report a failure from their Gitlab pipelines).
Compared to the existing check from gitlab-ci.yml, the Makefile check
differs in this respect:
- don't explicitly find *.py files: they are supposed to also be found
as a result of running 'file' on them;
- use git ls-tree instead of find: this is supopsedly faster as it
uses the index rather than readdir();
- don't output the count of warnings or errors: the output is a single
integer, which is confusing when there are errors, and even more so
when there are no, when it is simply '0';
- don't sort: the output is already stable and independent from the
locale;
- don't report the number of processed files: this information is
rather useless, and getting a hold of it would be more challenging
in this new code.
Note: ideally, we would want to use --null, --zero, or similar options,
with utilities that generates or parses a files listing. While git
ls-tree and xargs do support it, it becomes a little bit tricky to use
the --print0 option of file, and then grep in that output (it is not
undoable, but would requires replacing grep+cut with some sed trickery).
Since we do not expect our scripts names to contain funky chars (like
\n or a colon), we just hand-wave away that issue (and the old code was
doing the same assumption too).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As 18f6c26118 just did to silence the file lists commands, switch to
using $(Q) instead of a plain @, to silence the commands.
Using $(Q) will allow to debug the commands with V=1.
We keep @ for the calls to MESSAGE, though.
The commands that are not currently silenced are left as-is, and they
can be converted to being silent in a followup patch, if need be,
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
