For various reasons, like debugging or compliance, it is important to
identify what br2-external trees versions were used for a specific
build.
Add a Kconfig option that contains the version as computed by
support/scripts/setlocalversion; this will appear in the .config file
(but not in defconfig files, which is what we want).
Also generate that variable on the .mk side, so that it gets properly
exported in the environment, for post-build of post-iamge scripts to use
as they see fit (like, ensuring there is no dirtyness when in a CI for
example).
Reported-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the list of external trees is a private variable, but for
debugging or compliance, one may need to get that list.
Add a Kconfig option so that the list appears in the .config file, and
export the already existing .mk variable in the environment, so that
post-build or post-image scripts can use it.
Reported-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 8025cfad10)
[Peter: drop Makefile hunk]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 2dff6e93ca (package/readline: add upstream patch to fix crash
with invalid locale specification) fixed a regression in readline 8.2
[0], that could have been caught with a runtime test. readline is a
library, so we need an executable that exercises readline.
Since readline and bash are developped in tandem [1], it is only logical
to use bash to test readline.
Add a new runtime test for bash, that checks that we can indeed run an
interactive shell, and that an non-existing locale does not cause the
dreaded segfault. We do not use the default configuration, because it
uses a uclibc toolchain, and we want to reproduce against a glibc one.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1021109
[1] https://tiswww.case.edu/php/chet/readline/rltop.html#Bugs
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The past participle for "to fix" is "fix". The "did you forget" got
eluded into "forget", so again a past participle.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)
Expand the runtime test and the unit tests for check-package.
Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Provides python interface to database stored in hwdata
package. It allows you to get human readable description of
USB and PCI devices.
https://github.com/xsuchy/python-hwdata
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes:
support/testing/tests/package/test_shadow.py:55:1: W391 blank line at end of file
1 W391 blank line at end of file
make: *** [Makefile:1253: check-flake8] Error 123
https://gitlab.com/buildroot.org/buildroot/-/jobs/3918132888
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The kernel config board/qemu/aarch64-sbsa/linux.config has never been in
use by qemu_aarch64_sbsa_defconfig, neither via
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE, nor via
BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES.
test_edk2.py is using the kernel config
board/qemu/aarch64-sbsa/linux.config. However, storing a kernel config
that is not used by qemu_aarch64_sbsa_defconfig, in a directory that is
"owned" by qemu_aarch64_sbsa_defconfig, is bound to cause confusion.
Therefore, move the config file to a new subdirectory:
support/testing/tests/boot/test_edk2/
This is similar to how e.g. test_grub.py has a subdirectory:
support/testing/tests/boot/test_grub/
where it keeps the kernel config that is only used by test_grub.py.
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
ace needs FileHandle module:
Can't locate FileHandle.pm in @INC (you may need to install the FileHandle module) (@INC contains: /home/buildroot/autobuild/instance-0/output-1/host/lib/perl /usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /home/buildroot/autobuild/instance-0/output-1/build/ace-7.0.6/MPC/prj_install.pl line 17.
BEGIN failed--compilation aborted at /home/buildroot/autobuild/instance-0/output-1/build/ace-7.0.6/MPC/prj_install.pl line 17.
Fixes:
- http://autobuild.buildroot.org/results/9dee7c09fd7b41d276df0285a0f3dcae1a71f041
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 71ddf1a084)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fix the following build failure:
Can't locate object method "hexhash" via package "MD5" at utils/git-testament.pl line 47
Fixes:
- http://autobuild.buildroot.org/results/3dddcbbe7f6ecae5a2db6fac11fb659719452f73
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
shadow provides utilities to deal with user accounts.
The shadow package includes the necessary programs for converting UNIX
password files to the shadow password format, plus programs for managing
user and group accounts. Especially it is useful if rootless podman
container should be used, which requires newuidmap and newgidmap.
Co-authored-by: Nicolas Carrier <Nicolas.Carrier@orolia.com>
[Nicolas.Carrier@orolia.com provided the test case]
Signed-off-by: Raphael Pavlidis <raphael.pavlidis@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
I do constantly get mails that fluent-bit fails to build for s390x.
So added this to ensure that the s390x architecture is checked as well
if I manually do:
$ ./utils/test-pkg -p fluent-bit -a
Signed-off-by: Thomas Devoogdt <thomas.devoogdt@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
libjxl is the reference implementation of JPEG XL (encoder and decoder).
https://github.com/libjxl/libjxl
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Highway is a C++ library that provides portable SIMD/vector intrinsics.
https://github.com/google/highway
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In commit 04154a6517 (support/download/cargo-post-process: cargo
output for vendor config), we switched away from our hand-crafted
cargo.toml mangling, to use cargo itself to update that file.
In doing so, we enabled the shell pipefail option, so that we could
catch cargo failures, while redirecting its output through tee to the
cargo.toml.
However, pipefail is overzealous, and will hit us even for pipes we do
not want to globally fail, like the one that actually checks whether an
archive is already vendored or not:
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
...
with pipefail, the above may always fail:
- if the tarball is already vendored, grep will exit on the first
match because of -q (it only needs a single match to decide that its
return code will be zero), so the | will get closed, and tar may
get -EPIPE before it had a chance to finish listing the archive, and
thus would terminate in error;
- if the tarball is not vendored, grep will exit in error.
It turns out that the tee was only added so that we could see the
messages emitted by cargo, and still fill the cargo.tom with the output
of cargo.
But that's a bit overkill: the cargo messages are going to stderr, and
the blurb to add to cargo.toml to stdout, so we just need to redirect
stdout.
Yes, we do not see what cargo added to cargo.toml, but that is not so
interesting.
Still, cargo ends its messages with a suggestion for the user to modify
cargo.toml, with:
To use vendored sources, add this to your .cargo/config.toml for this project:
But since we've already redirected that to cargo.toml, there is nothing
for the user to edit, so the above can get confusing. Emit a little
blurb that states that everything is under control.
And then we can drop pipefail.
Note: the go-post-process initially had pipefail too, but it was dropped
in bfd1a31d0e (support/download/go-post-process: drop -o pipefail) as
it was causing spurious breakage when extracting the archive before
vendoring, so it is only reasonable that we also remove it from the
cargo-post-process.
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/12a/12a63ae177fe3ed0c9a1ef2fa01870f334f36b0f/
Currently, when the post-process helper fails while downloading from
upstream, there is no fallback to the backup mirror.
In case the post-process helper fails, we must consider that to be a
download failure, so we must bail out as if the download backend itself
did fail, but we fail to do so.
Duplicate the logic we have for the download helper: if the post-process
helper fails, remove the downloaded stuff, and continue on to the next
URI, which will ultimately hit the backup mirror (if one has been
configured).
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/820/820e98b1c126469b1f180f078d102ded43b9c40e/
scripts/Makefile.am of mosh-1.4.0 needs the perl diagnostics module on the host:
make[3]: Entering directory '/home/buildroot/autobuild/instance-2/output-1/build/mosh-1.4.0/scripts'
perl -Mdiagnostics -c ./mosh.pl
Can't locate diagnostics.pm in @INC (you may need to install the diagnostics module) (@INC contains: /home/buildroot/autobuild/instance-2/output-1/host/lib/perl /usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5).
BEGIN failed--compilation aborted.
So add a check for it in dependencies.sh similar to the other perl modules.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit adds two new test cases:
- TestNodeJSBasic which builds a target configuration with just
NodeJS enabled, and which runs a very simple NodeJS script on the
target.
- TestNodeJSModule, which builds a target configuration with NodeJS
enabled + the installation of one extra module, which means npm on
the host (from host-nodejs) is used, and which runs a very simple
NodeJS script on the target that uses this extra module.
Having both tests separately allows to validate that both nodejs-only
and nodejs+host-nodejs configurations behave correctly, at least in
minimal scenarios.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).
Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.
In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.
Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.
Apply these checks for shell scripts:
- shellcheck;
- trailing space;
- consecutive empty lines;
- empty last line on file;
- newline at end of file.
Update the list of ignored warnings.
Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
The go vendoring fails on CentOS 7 (which uses git 1.8.3.1) with errors
related to shallow clones:
make docker-compose-source
..
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4
github.com/docker/compose/v2/pkg/mocks imports
github.com/theupdateframework/notary/client imports
github.com/docker/go/canonical/json: github.com/docker/go@v1.5.1-1.0.20160303222718-d30aec9fd63c: invalid pseudo-version: git fetch --unshallow -f origin in /home/jacmet/source/buildroot-mirror/output/host/share/go-path/pkg/mod/cache/vcs/48fbd2dfabec81f4c93170677bfc89087d4bec07a2d08f6ca5ce3d17962677ee: exit status 128:
fatal: git fetch-pack: expected shallow list
make[1]: *** [/home/jacmet/source/buildroot-mirror/output/build/docker-compose-2.15.1/.stamp_downloaded] Error 1
It works with git 2.0.0 (released May 2014, included in Debian 8), so check
for >= 2.0.0 with logic similar to the GNU patch version check.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation.
It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.
The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).
Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).
[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The gcc plugin test was not using the -q option to grep causing it to print
the line to stdout, so fix that.
While we're at it, adjust the locale check to use grep -q instead of
redirecting to /dev/null for consistency with the other checks.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
snapshot.debian.org is notoriously slow, and quite often leads to
timeouts when downloading packages. To give us a better chance of
succeeding, let's retry 3 times before failing.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
SPAKE2 password-authenticated key exchange (in pure python).
This library implements the SPAKE2 password-authenticated key
exchange ("PAKE") algorithm. This allows two parties, who share a
weak password, to safely derive a strong shared secret (and
therefore build an encrypted+authenticated channel).
https://github.com/warner/python-spake2
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 4cdd99190e (support/dependencies/dependencies.sh: require open perl
package for libxcrypt) added a check for the "open" perl module for
libxcrypt, but it does not work as "open" cannot be directly used with
"require" as an argument is needed:
perl -e "require open"
Not enough arguments for open at -e line 1, at EOF
Execution of -e aborted due to compilation errors.
So special case the check to instead check with "use open ':std'".
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/16c/16cba783be11cc5024f9c56bb0a7abb0acf13ef8/
The configure script of libxcrypt needs the open perl module on the host:
Can't locate open.pm in @INC (you may need to install the open module) (@INC
contains: /home/buildroot/autobuild/instance-2/output-1/host/lib/perl
/usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5
/usr/share/perl5) at ./build-aux/scripts/expand-selected-hashes line 20.
So add a check for it in dependencies.sh similar to how it is done for mpv.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit d631615eb1 (support/testing: test check-package ignore list) added
a too long line causing the check-flake8 target to fail:
support/testing/tests/utils/test_check_package.py:233:133:
E501 line too long (138 > 132 characters)
https://gitlab.com/buildroot.org/buildroot/-/jobs/3726245521
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes http://autobuild.buildroot.net/results/166/1667b4ebd8e16e7d8b47bc3ca128a76daf27b4d9/
The configure script of host-libxml-parser-perl needs the English perl
module on the host:
ERROR from evaluation of
/home/buildroot/autobuild/instance-1/output-1/build/host-libxml-parser-perl-2.46/Expat/Makefile.PL:
Can't locate English.pm in @INC (you may need to install the English module)
(@INC contains:
/home/buildroot/autobuild/instance-1/output-1/build/host-libxml-parser-perl-2.46/inc
/home/buildroot/autobuild/instance-1/output-1/host/lib/perl
/usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5
/usr/share/perl5 .) at ./Makefile.PL line 3.
So add a check for it in dependencies.sh similar to the existing check for
ExtUtils::MakeMaker.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running tests with top-level parallel builds can speed up running some
tests, expecially those that have a lot of packages like the systemd
init tests.
Trigger TLPB when the configuration enables per-package directories.
We're using the jlevel argument, which normally is used for BR2_JLEVEL
as the value for calling make -j<N> at the top-level. In fact,
BR2_JLEVEL is "unused" when using TLPB, because the top-level make
acts as the job server that distributes tokens to sub-makes (except
for the few build systems like waf or scons that don't support this),
so it's really the top-level make -j<N> that determines the level of
parallelism, and BR2_JLEVEL doesn't really have an effect.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Thomas: extend explanation a bit]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Start counting the line numbers in 1 instead of 0, in case an error
must be printed.
Both the error about a developer entry with no file entry and the error
about a file entry with no developer entry actually belong to the
non-empty line previous the one being analysed, so in these cases print
the line number from the line before.
Also count empty and comment lines, so a developer fixing the file can
jump to the correct line (or the nearest one).
At same time standardize the messages, printing the line number
also in the case of a warning for a file that is not in the tree
anymore.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
