Extend docker_test() to expose a random (8888) port to verify that doesn't
fail, and extend the docker-compose test to run the busybox httpd in the
background, expose that as port 80 and verify that /etc/resolv.conf could be
fetched by wget.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The python2 support has been removed since the python-colorzero bump version to 2.0.
[1] 73bf3292e1
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The python2 support has been removed since the python-colorzero bump version to 2.0.
Remove the gpiozero test with python2
[1] 73bf3292e1
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 76b4f9e9b6)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Create a new user 'sudotest' to validate that sudo really works (i.e.
properly has setuid).
Creating the user and adding it to sudoers is done at runtime, otherwise
we'd need to add extra files to the config which complicates things a
little bit.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Kernel 4.16.7 is old enough to produce the "multiple definition of `yylloc'"
error which is fixed in newer versions.
Bump the test kernel version from 4.16.7 to 5.10.34 to prevent this error wwhen
building the test image.
Signed-off-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Test that the TAICLOCK and TCP servers are working.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Arnout: indent config lines more]
Test that s6-rc service database compilation is working.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Arnout: indent config lines more]
Test that a few basis utilities are working.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Arnout: indent config lines more]
Test that directory scanning and supervision is working.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Arnout: properly indent, and use textwrap to dedent again.]
Test that the interpreter can run a basic command.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Arnout: indent config lines more]
Commit 54d3d94b6e broke the 'hg' download
method, in a similar way as it broke the 'git' download method (later fixed
with commit b70ce56651), by introducing extra
output on stdout in a case where the output is redirected.
In the case of 'hg', the 'hg archive' step uses shell redirection rather
than directly letting hg write the output file, since commit
76b51f90c0.
As a result, the extra print added by the _hg function is prepended to the
actual archive, causing an invalid archive.
Fix by using the _plain_hg function instead. The disadvantage is that the
command for 'hg archive' is no longer printed.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
ijson < 2.5 (as available in Debian 10) use the slow python backend by
default instead of the most efficient one available like modern ijson
versions, significantly slowing down cve checking. E.G.:
time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total
To
93,53s user 2,00s system 98% cpu 1:36,65 total
E.G. almost 2x as fast.
As a workaround, detect when the python backend is used and try to use a
more efficient one instead. Use the yajl2_cffi backend as recommended by
upstream, as it is most likely to work, and print a warning (and continue)
if we fail to load it.
The detection is slightly complicated by the fact that ijson.backends used
to be a reference to a backend module, but is nowadays a string (without the
ijson.backends prefix).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
OpenZFS is an advanced file system and volume manager which was originally
developed for Solaris and is now maintained by the OpenZFS community. This
repository contains the code for running OpenZFS on Linux and FreeBSD.
http://zfsonlinux.org/
Signed-off-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
[me:
- fix test case on how to use a pre-built toolchain
- reorder the test case config
- add test case with glibc
- drop superflous test timeout override
- only select libtirpc when C library lacks native RPC
- drop unused ZFS_MODULES variable
- drop ZFS_CPE_ID_PREFIX and ZFS_AUTORECONF_OPTS which are defaults
- drop NLS options, already set in a generic manner
- drop incomplete/improper sysvinit support
- some cosmetics
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit bb10b0dfe6)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When a --transform expression is provided, it is by default also applied
to the target of a symlink.
When we create tarballs (from git or svn checkouts), we use a --transform
expression to replace the leading ./ with the package name and version.
This causes issues when a package contains symlinks that points to
./something, as the leading './' is also replaced.
Fix that by using the 'S' transformation scope flag, as described in the
tar manual:
https://www.gnu.org/software/tar/manual/html_node/transform.html#transform
In addition, several transformation scope flags are supported, that
control to what files transformations apply. These are:
‘r’ Apply transformation to regular archive members.
‘R’ Do not apply transformation to regular archive members.
‘s’ Apply transformation to symbolic link targets.
‘S’ Do not apply transformation to symbolic link targets.
‘h’ Apply transformation to hard link targets.
‘H’ Do not apply transformation to hard link targets.
Default is ‘rsh’ [...].
Fixes: #13616
This has been checked to not change any of the existing hash for any of
our git-downloaded package (some are host-only, hence the few fixups):
---8<---
$ m="$( git grep -l -E -- -br[[:digit:]]+.tar.gz boot package/ \
|awk -F/ '{print $(NF-1)}' \
|sed -r -e 's/(imx-mkimage|netsurf-buildsystem|prelink-cross|qoriq-rcw|vboot-utils)/host-\1/g' \
-e 's/$/-source/'
)"
$ make defconfig; make clean; BR2_DL_DIR=$(pwd)/trash-me make ${m}
---8<---
Note: it is unclear what the 'H' flag does nor how it works, because the
concept of "target of a hardlink" is not obvious; probably it has to do
with how tar internally detects and stores hardlinks. Since we do not
yet have any issue with hardlinks, just ignore the problem for now, and
postpone until we have an actual issue with a real test-case.
Signed-off-by: Jean-pierre Cartal <jpcartal@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
- re-indent commit log
- add scriptlet to test existing hashes
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 54d3d94b6e ("support/download: print
command used for download") broke the git and svn download helpers, because
these helpers have invocations of the _git/_svn commands where the exact
output matters.
For example for git, this would result in:
date: invalid date ‘GIT_DIR=.../dl/libyuv/git/.git git log -1 --pretty=format:%ci \n2019-04-12 17:48:45 +0000’
Detected a corrupted git cache.
Removing it and starting afresh.
Fix by splitting the _git function in two: _git and _plain_git.
The former echoes the command, and then calls the latter.
Most invocations use _git as before, but those cases where the output should
not be disturbed, directly call _plain_git.
For symmetry, all download helpers are aligned, even though only the git and
svn helpers were broken.
Fixes: #13631
Fixes:
http://autobuild.buildroot.org/results/c2f/c2fcd4aa6660e3c2f9c6f85646ca7dfe0db56040/
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: add bug report and autobuild failure]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Even though that most download commands actually print some output, like
progress indication or other messages, the actual command used is not. This
makes it hard to analyze a build log when you are not fully familiar with
the typical output of said log.
Update the download helpers to do just that, respecting any quiet/verbose
flag so that a silent make (make -s) does not get more verbose.
Note: getting rid of the duplication of the command in the script is not
straightforward without breaking support for arguments with spaces.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: use printf, not echo]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Most 'verbose' variable inside the download helpers actually mean 'quiet'.
I.e. they are assigned in case quiet operation is requested, and empty in
case of non-quiet operation. Using the name 'verbose' for such a variable is
confusing, especially when you want to test the variable on emptiness or
non-emptiness (in a subsequent commit).
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In Makefiles, variables are split, filtered, and otherwise mangled on
a space as a separator. In a shell, they will also be split on TABs.
We split and filter and iterate on variables in a lot of places, and
most importantly, spaces in PATH is very seldom tested, if at all, so
a lot of packages will not be working properly in such a situation.
For example, the config.guess contains constructs that are not resilient
to a space in PATH:
PATH=$PATH:/.attbin ; export PATH
Also, our fakedate will iterate over PATH:
for P in `echo $PATH | tr ':' ' '`; do
Those are only two cases, but the first means basically all
autotools-based packages are susceptible to subtle breakage.
Furthermore, Buildroot itself does not support that the top-level or
output directories are in a path with spaces anyway.
So, instead of chasing all cases that might be potentially broken,
let's just detect the case and bail out, like we already do when PATH
contains a \n, or when it contains the current working directory.
Reported-by: Dan Raymond <draymond@foxvalley.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since c043ecb20c (support/download: change format of archives
generated from svn), the svn backend uses the generic helper to
create reproducible archives.
That helper really does its job as expected, but the svn backend
is flawed in two ways:
- the first, most obvious breakage happens with versions older
than 1.9, as they do not support the '--show-item' option
for the 'info' action;
- the second is more involved, in that svn will by default
expand the old, legacy, deprecated, cumbersome CVS-style
keywords, in the form of revision marks like '$Date$' in a
C-style comment in a source file. These replacements are
done on checkout as well as on export, and they use local
settings, like the local locale and timezone.
This means that two people with different settings, will get
different sources when the svn-checkout or svn-export the same
revision from the same tree...
Needless to say that this is not very reproducible...
While the first is easily solved, the second is more involved.
We need to ensure that what source is used initially to compute
the hash, will also be the source that are used to check the hash.
There are basically two solutions:
1. we ensure the same environment, by forcing the timezone and
the locale to arbitrary values
2. we disable keyword expansion
For the first solution, this still leaves the possibility that we
miss some environment settings that have an impact on the keyword
expansion. It would mean that Yann's settings be used, as he did
introduce the hash for the only svn-downloaded package we have,
avrdude, settings which are:
TZ=Europe/Paris
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="fr_FR.utf8"
LC_NUMERIC="fr_FR.utf8"
The second option means that the generated archives change. That
means we'd have to bump the archive version for svn downloads, and
that we update the hashes for all the svn-downloaded packages.
We chose to go with the second option, because this is what really
makes more sense, rather than hard-coding arbitrary values in the
environment. And we also have only one svn-downloaded package,
avrdude.
And thus, we're reaching the trigger for this change: avrdude is
impacted by the CVS-keyword expansion issue:
https://svn.savannah.gnu.org/viewvc/avrdude/trunk/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js?revision=1396&view=markup
which would give two different files when checked out on different
machines:
diff -durN foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js
--- foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200
+++ bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200
@@ -1,6 +1,6 @@
/**
* @preserve jquery.layout 1.3.0 - Release Candidate 30.51
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
* $Rev: 303005 $
*
* Copyright (c) 2012
@@ -4718,7 +4718,7 @@
/**
* jquery.layout.state 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2010
* Kevin Dalman (http://allpro.net)
@@ -5074,7 +5074,7 @@
/**
* jquery.layout.buttons 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2010
* Kevin Dalman (http://allpro.net)
@@ -5356,7 +5356,7 @@
/**
* jquery.layout.browserZoom 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2012
* Kevin Dalman (http://allpro.net)
So we also update the hash for avrdude.
Fixes:
http://autobuild.buildroot.org/results/e3b/e3b0508047f32008ebfa83c5255ec5994b6af120/ (time issue)
http://autobuild.buildroot.org/results/48e/48e78e84b425e79cdb98c16ab40247a0fa7e9676/ (keyword expansion issue)
Reported-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Alexander Sverdlin <alexander.sverdlin@gmail.com>
Reviewed-by: Alexander Sverdlin <alexander.sverdlin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
FOO_CPE_ID_VALID really ought to be an internal implementaion detail.
Packages that really want to trigger their CPE defintitions really
should set one of the actual variables to a meaningful value.
There are two CPE-related variables that we could chose to set to
replace FOO_CPE_ID_VALID: FOO_CPE_ID_VENDOR and FOO_CPE_ID_PRODUCT.
Between those two, _VENDOR more often diverges from the default than
_PRODUCT does, so that's what we use.
---8<------8<------8<------8<------8<---
#!/bin/bash
# Replace FOO_CPE_ID_VALID = YES with FOO_CPE_ID_VENDOR = foo_project
for i in $(git grep -l -E '[^)]_CPE_ID_VALID = YES' package support); do
pkg="$(basename "${i%/*}")"
sed -r -i -e "s/_CPE_ID_VALID = YES/_CPE_ID_VENDOR = ${pkg}_project/" "${i}"
done
---8<------8<------8<------8<------8<---
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[Peter: update cpe-test comment to reflect pkg3 change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some download backends, like svn, will provide timestamps with a
sub-second precision, e.g.
$ svn info --show-item last-changed-date [...]
2021-02-19T20:22:34.889717Z
However, the PAX headers do not accept sub-second precision, leading to
failure to download from subversion:
tar: Time stamp is out of allowed range
tar: Exiting with failure status due to previous errors
make[1]: *** [package/pkg-generic.mk:148: [...]/build/subversion-1886712/.stamp_downloaded] Error 1
Fix that by massaging the timestamp to drop the sub-second part. We
do that in the generic helper, rather than the svn backend, so that
all callers to the generic helper benefit from this, as this is more
an internal details of the tarball limitations, than of the backends
themselves.
Reported-by: Roosen Henri <Henri.Roosen@ginzinger.com>
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
- add Henri as reporter
- move it out of the svn backend, and to the generic helper
- reword the commit log accordingly
- use an explicit time format rather than -Iseconds
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
As reported on IRC by sephthir, the gitlab test of the defconfig
qemu_sparc_ss10_defconfig doesn't error out while the system
is not working properly.
This is because we explicitly wait for the timeout as an expected
condition, but do not check for it. Indeed, pexpect.expect() returns
the index of the matching condition in the list of expected conditions,
but we just ignore the return code, so we are not able to differentiate
between a successful login (or prompt) from a timeout.
By default, pexepect.expect() raises the pexpect.TIMEOUT exception on a
timeout, and we are already prepared to catch and handle that exception.
But because pexpect.TIMEOUT is passed as an expected condition, the
exception is not raised.
Remove pexpect.TIMEOUT from the list of expected conditions, so that the
exception is properly raised again, and so that we can catch it.
The qemu_sparc_ss10_defconfig is already fixed by
4d16e6f532.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
[yann.morin.1998@free.fr: reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 610e67b1fc)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add the list of <pkg>_IGNORE_CVES to the json output to show that we have a
known cause (available patch or the CVE is not valid for our package
configuration) that a affected CVE is not reported.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
pickle is no longer used since 09a71e6a75
Fixes:
support/scripts/cpedb.py:7:1: F401 'pickle' imported but unused
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the CPE XML database is parsed into a Python dict, which is
then pickled into a local file, to speed up the processing of further
invocations.
However, it turns out that since the initial implementation, we have
switched the XML parsing from the out of tree xmltodict module to the
standard ElementTree one, which has made the parsing much faster. The
pickle caching only saves 6 seconds, on something that takes more than
13 minutes total.
In addition, this pickle caching consumes a significant amount of RAM,
causing the Python process to be OOM-killed on a server with 4 GB of
RAM.
So let's just drop this caching entirely.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 64/musl, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 64/glibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Most of the toolchains now use gcc 9.x and kernel headers 5.9, instead
of gcc 8.x and kernel headers 5.4.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test Xtensa/uclibc, use a pre-built Bootlin toolchain.
To be noted: that fragment was in fact already using a Bootlin
bleeding-edge toolchain, because BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y is
missing from the fragment:
$ cat support/config-fragments/autobuild/br-xtensa-full.config >.config
$ make olddefconfig
$ grep BOOTLIN .config
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_XTENSA_LX60_UCLIBC_BLEEDING_EDGE=y
# BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_XTENSA_LX60_UCLIBC_STABLE is not set
The original fragment was supposed to use a stable toolchain, so we
switch to explictly use a stable Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- add blurb about missing BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test x86-64/musl, use a pre-built Bootlin toolchain.
The previous configuration was for an Atom platform, but the Bootlin
toolchains only provide a Core i7 configuration. Since this is close
enough, we change to use this Core i7 configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test x86-64/uclibc, use a pre-built Bootlin toolchain.
The previous configuration was for Core2 platform, but the Bootlin
toolchains only provide a Core i7 configuration. Since this is close
enough, we change to use this Core i7 configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test SPARC64/glibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test SPARC/uclibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test SH4/uclibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: drop BR2_sh4=y which is the default]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 32/glibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test PowerPC e500mc/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test PowerPC64le Power8/glibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test OpenRISC/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test nios2/glibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test mipsel/uclibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test mipsel32r6/glibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test Microblaze EL/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test m68k 5208/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test m68k 68040/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv7-M/uclibc, use a pre-built Bootlin
toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv7/musl, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test ARM Cortex-A9/glibc, use a pre-built Bootlin
toolchain. Since this was meant to test very recent version of
toolchain components, we use the bleeding edge toolchain variant.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv5/uclibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test ARCle HS38/uclibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of using an external toolchain built specifically for the
autobuilders to test AArch64/glibc, use a pre-built Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This commit extends pkg-stats to leverage the recently introduced
CPEDB class to verify that the CPEs provided by Buildroot packages are
indeed known in the official CPE dictionnary provided by NVD.
Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
There is no need to get both the key and the value out of the dict if the
key is not used, so use dict.values() instead.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In commit 7a607dab33
("support/scripts/pkg-stats: support generating stats based on
configured packages"), we added a -c option to pkg-stats to generate a
report based on the list of packages enabled in the configuration,
rather than for all packages.
This is done based on the list of packages returned in JSON format by
"make show-info". However, we use the keys of the JSON dict returned
by "make show-info", which include the host- prefix of host
packages. Due to this, none of the host packages are currently
matching and therefore they are not reported in the pkg-stats -c
output.
This commit fixes that by using the recently introduced "name"
property in the "make show-info" JSON dict.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use anonymous '_' for unused variable]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 551cb63007)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The --cpeid option was mistakenly introduced by commit
92e7089a8c ("support/script/pkg-stats:
show CPE ID in results") but is in fact not necessary.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since commit bd665d182c
("support/scripts/pkg-stats: improve rendering of CVE information"),
we have better reporting of CVE related information, based on
pkg.status['cve']. However, this commit broke pkg-stats when the
--nvd-path option is not passed, and therefore no CVE information is
available.
This commit fixes that, by making use of the is_status_ok(),
is_status_error() and is_status_na() methods recently introduced.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Make is_status_ok() work when the given status name is not even listed
in the status dict. This will be necessary for following commits.
Introduced similar methods for the error and na status, which will be
used in following commits.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.
parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.
This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.
Before this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
"name": "stm32f469_disco",
"path": "configs/stm32f469_disco_defconfig",
"developers": []
}
After this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
"Floris Bos <bos@je-eigen-domein.nl>",
"Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
"name": "stm32f469_disco",
"path": "configs/stm32f469_disco_defconfig",
"developers": [
"Christophe Priouzeau <christophe.priouzeau@st.com>"
]
}
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
With Python 3.8, the following deprecation warnings are emitted:
/home/thomas/projets/buildroot/./support/scripts/pkg-stats:418: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
/home/thomas/projets/buildroot/./support/scripts/pkg-stats:536: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
The correct way to pass coroutines is to use asyncio.create_task(),
but this is rather new method (Python 3.7), and using it breaks
compatibility with older Python versions. As suggested at
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task,
use the more cryptic, but also more compatible asyncio.ensure_future()
method.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The cve module needs ijson, which may not be installed. Since cve
matching is only enabled when --nvd-path is passed, it is a bit silly
to error out about ijson being missing if it's not used.
So instead of unconditionally importing the cve module, only do it
conditionally.
However, instead of doing it right at the point where it is used, we
do it at the beginning of the main() function. Indeed, if the cve
module is needed but cannot be imported, we want to error out
immediately rather than doing a whole bunch of things, and failing on
the user later on in the middle of the pkg-stats execution.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Until now the bootlin-x86-64-glibc was using a Bootlin toolchain as a
custom external toolchain. However, now that we have the
toolchain-external-bootlin package explicitly supporting Bootlin
toolchains as known toolchain profiles, it makes sense to use
that. Indeed, this will ensure that this autobuilder configuration
will use the latest available version of the Buildroot toolchain for
x86-64 glibc.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Fix U-Boot config snippet in TestATFAllwinner. Bump U-Boot
version to fix DTC build on hosts with gcc 10 and add
pylibfdt dependency.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
There were only two users of Marvell ATF: SolidRun MacchiatoBin board
and ClearFrog GT 8k board. After mv-ddr-marvell package update both
boards switched to upstream ATF. Remove tests for now unused
Marvell ATF.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
So far, we checked that the tar present on the host was at most tar
1.29, because tar 1.30 changed the way it generates archives.
Having a maximum tar version requirement meant that we would eventually
always have to build our own host-tar, as distributions are updating
the version they use.
But now, we have found a way to generate reproducible archives starting
with tar 1.27 onward, so we no longer need the check for a maximum tar
version, so we can drop that requirement.
Note: this is semantically a revert of b8fa273d50 (check-host-tar.sh:
blacklist tar 1.30+), but keeping the new, mostly-linear code-path.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Like we recently did for git, switch the archives generated from
subversion to be reproducible whatever the tar version.
We have no in-tree users of the svn backend which also has hashes,
so no hash to update.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Commit 89f5e9893 (support/download/svn: generate reproducible svn
archives) did what it said, but can be siplified a bit.
Indeed, we are doing an svn export, so we won't have any of the .svn
directories, neither at the root of the extract, nor in any of the
sub-directories.
As such, we do not need to filter them out when we generate the list
of files to include in the archive.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Switch to using the tarball helper, that can generate reproducible
archives whatever the tar version >= 1.27.
However, those archives are not identical to the previous ones generated
in the (now-broken) gnu format.
To avoid any clashing between old and new archives, and new and old
Buildroot versions, we need to name the new generated archives
differently from the existing ones.
So, we bump the git-specific format-version to -br1.
The %ci date has been supported by git back to 1.6.0, released August
2008); it is not strictly ISO8601, but is still accepted as a PAX date
header. The strict ISO8601 placeholder, %cI, was only introduced with
2.2.0, release in November 2014, so too recent to be widely available.
As the format and the names of the archives changes, we need to update
all the hash files with the new names and hashes.
Of all the bootloaders that have a git download method, vexpress-firmware
is the only one to have a hash. Others have no hash files, or they have
explicitly set BR_NO_CHECK_HASH_FOR.
For the packages, linux-headers is the special snowflake, as the git
download is only for custom git tree, so it is excluded from the hash
verification with BR_NO_CHECK_HASH_FOR.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
---8<------8<------8<------8<---
#!/bin/sh
# Find and download all packages using git as backend.
# Manually fix hashes for affected packages.
# Packages that only have a host variant
HOST_ONLY='imx-mkimage|mxsldr|netsurf-buildsystem|opkg-utils|prelink-cross|qoriq-rcw|vboot-utils'
# Packages that have a non-git main _SOURCE, and/or which
# have BR_NO_CHECK_HASH_FOR for the git _SOURCE
NOT_GIT='aufs|aufs-util|xenomai|linux-headers'
export BR2_DL_DIR=$(pwd)/temp-dl-dir
make defconfig
make $( git grep -l -E 'SITE_METHOD[[:space:]]*:?=[[:space:]]*git\>|_SITE[[:space:]]*:?=[[:space:]]*git:' \
boot/vexpress-firmware/ package/ \
|sed -r -e 's,.*/([^/]+)\.mk,\1,' \
|sed -r -e '/^('"${NOT_GIT}"')$/d;' \
-e 's/^('"${HOST_ONLY}"')/host-\1/;' \
-e 's/$/-legal-info/;'
)
---8<------8<------8<------8<---
We currently need to generate reproducible archives in at least two
locations: the git and svn download backends. We also know of some
future potential use (e.g. the other download backends, like cvs, or
in the upcoming download post-processors for vendoring, like cargo
and go).
However, we are currently limited to a narrow range of tar versions
that we support, to create reproducible archives, because the gnu
format we use has changed with tar 1.30.
As a consequence, and as time advances, more and more distros are,
or will eventually start, shipping with tar 1.30 or later, and thus
we need to always build our on host-tar.
Now, thanks to some grunt work by Vincent, we have a set of options
that we can pass tar, to generate reproducible archives back from
tar-1.27 and up through tar-1.32, the latest released version.
However, those options are non-trivial, so we do not want to have
to repeat those (and maintain them) in multiple locations.
Introduce a helper that can generate a reproducible archive from
an input directory.
The --pax-option, to set specific PAX headers, does not accept
RFC2822 timestamps which value are too away from some fixed point
(set atcompile-time?):
tar: Time stamp is out of allowed range
However, the same timestamps passed as strict compliant ISO 8601 are
accepted, so that's what we expect as a date format.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
---8<------8<------8<------8<---
# Here is a Makefile used to test all the versions of tar, with
# different output formats and different sets of options:
# Versions prior to 1.27 do not build on recent machines, because
# 'gets()' got removed (rightfully so), so don't count them as
# candidates.
VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32
DATE = Thu 21 May 2020 06:44:11 PM CEST
TARS = \
$(patsubst %,test_gnu_%.tar,$(VERSIONS)) \
$(patsubst %,test_posix_%.tar,$(VERSIONS)) \
$(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS))
all: $(TARS)
sha1sum $(^)
.INTERMEDIATE: test_%.tar
test_gnu_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=gnu \
-T list \
>$(@)
test_posix_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=posix \
-T list \
>$(@)
test_posix_paxoption_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=posix \
--pax-option='delete=atime,delete=ctime,delete=mtime' \
--pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \
-T list \
>$(@)
list: .FORCE
list: test
(cd test && find . -not -type d ) |LC_ALL=C sort >$(@)
LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng
test: .FORCE
test:
rm -rf test
mkdir -p test/bar
echo foo >test/Foo
echo bar >test/bar/Bar
ln -s bar/Bar test/buz
echo long >test/Very-$(LONG)-filename
ln test/Very-$(LONG)-filename \
test/short
.PRECIOUS: tar.%
tar.%: tar-%
cd $(<) && ./configure
$(MAKE) -C $(<)
install -m 0755 $(<)/src/tar $(@)
.PRECIOUS: tar-%
tar-%: tar-%.tar.gz
tar xzf $(<)
.PRECIOUS: tar-%.tar.gz
tar-%.tar.gz:
wget "https://ftp.gnu.org/gnu/tar/$(@)"
.FORCE:
clean:
rm -rf tar-* tar.* test_* test list
---8<------8<------8<------8<---
support/scripts/pkg-stats:81:22: E211 whitespace before '('
support/scripts/pkg-stats:404:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/pkg-stats:561:12: E713 test for membership should be 'not in'
support/scripts/pkg-stats:567:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:595:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1051:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1057:1: E302 expected 2 blank lines, found 1
Also fix:
support/scripts/pkg-stats:1054:5: E722 do not use bare 'except'
found by a more recent flake8 version. The exception may be either
IndexError or AttributeError, so use Exception to catch either.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This commit improves pkg-stats to fill in pkg.status['cve'] depending
on the situation for CVEs affecting this package. They are then used
in the HTML rendering.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Virtual packages (with in pkg-stats speak have "no valid
infrastructure") and packages that have no version specified cannot be
used for CVE checking. They trigger a bunch of warnings from the CVE
checking code, as it cannot parse their version: they don't have any
version. So instead, we simply skip those packages.
A follow-up commit will improve the reporting to be able to
distinguish those packages from packages that have seen their CVEs
checked and don't have any reported.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit modifies cve.py, as well as its users cve-checker and
pkg-stats to support CPE ID based matching, for packages that have CPE
ID information.
One of the non-trivial thing is that we can't simply iterate over all
CVEs, and then iterate over all our packages to see which packages
have CPE ID information that match the CPEs affected by the
CVE. Indeed, this is an O(n^2) operation.
So instead, we do a pre-filtering of packages potentially affected. In
check_package_cves(), we build a cpe_product_pkgs dict that associates
a CPE product name to the packages that have this CPE product
name. The CPE product name is either derived from the CPE information
provided by the package if available, and otherwise we use the package
name, which is what was used prior to this patch.
And then, when we look at CVEs, we only consider the packages that
have a CPE product name matching the CPE products affected by the
CVEs. This is done in check_package_cve_affects().
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit improves the pkg-stats script to show the CPE ID of
packages, if available. For now, it doesn't use CPE IDs to match CVEs.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The logic in gen-bootlin-toolchains was assuming all glibc toolchains
have RPC support, which is no longer true since glibc 2.32 has dropped
RPC support.
It turns out that gen-bootlin-toolchains already had some proper logic
that selects BR2_TOOLCHAIN_HAS_NATIVE_RPC depending on the presence of
BR2_TOOLCHAIN_EXTERNAL_INET_RPC in the toolchain fragment. As such
toolchain fragments have been fixed in https://toolchains.bootlin.com,
we can now rely on this to properly decide if the toolchain has RPC
support or not.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 804a9e1865)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
quazip requires cmake 3.15 since version 1.0 and
89e7c201f0818adc8224
The rationale for this requirement is that "default locations for the
install(TARGETS command based on the GNUInstallDirs package were only
added in 3.14" and "3.15 is not that much of a difference from 3.14 and
it introduced a lot of useful UI improvements.":
https://github.com/stachenov/quazip/issues/82
Fixes:
- http://autobuild.buildroot.org/results/5d848a46109aef448ea1d1b857a500d9461dc2d9
Note: we also have some patches to allow some packages to build with
cmake-3.10, and this will not be tenable over the long run.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[yann.morin.1998@free.fr: add the "note"]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When boot-qemu-image.py script was added, we wanted to run
each qemu defconfig in gitlab, so we expect that all qemu
defconfig generate the script start-qemu.sh in images
directory.
Don't make it a hard requirement even if we prefer to be
able to do a runtime test for each qemu defconfig.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds a number of test cases to verify that the CPE_ID_*
variables are properly handled by the generic package infrastructure
and that the "make show-info" JSON output matches what we expect.
A total of 5 different example packages are used to exercise different
scenarios of CPE_ID_* variables usage.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the version encoded in a CPE is '-', we assume all
versions are affected, but when it's '*' with no further range
information, we assume no version is affected.
This doesn't make sense, so instead, we handle '*' and '-' in the same
way. If there's no version information available in the CVE CPE ID, we
assume all versions are affected.
This increases quite a bit the number of CVEs and package affected:
- "total-cves": 302,
- "pkg-cves": 100,
+ "total-cves": 597,
+ "pkg-cves": 135,
For example, CVE-2007-4476 has a CPE ID of:
cpe:2.3🅰️gnu:tar:*:*:*:*:*:*:*:*
So it should be taken into account. In this specific case, it is
combined with an AND with CPE ID
cpe:2.3⭕suse:suse_linux:10:*:enterprise_server:*:*:*:*:* but since
we don't support this kind of matching, we'd better be on the safe
side, and report this CVE as affecting tar, do an analysis of the CVE
impact, and document it in TAR_IGNORE_CVES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We should not rely on host installed bison/flex for target code. This
ensures better reproducibility of generated code.
http://lists.busybox.net/pipermail/buildroot/2020-November/296786.html
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that pkg-stats is able to generate its output based on the list of
packages enabled in the current configuration, cve-checker doesn't
serve any purpose.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
pkg-stats was initially a Buildroot maintenance oriented tool: it was
designed to examine all Buildroot packages and provide
statistics/details about them.
However, it turns out that a number of details provided by pkg-stats,
especially CVEs, are relevant also for Buildroot users, who would like
to check regularly if their specific Buildroot configuration is
affected by CVEs or not, and possibly check if all packages have
license information, license files, etc.
The cve-checker script was recently introduced to provide an output
relatively similar to pkg-stats, but focused on CVEs only.
But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses
"make show-info" to only consider packages enabled in the current
configuration.
So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
to generate its output based on the list of configured packages. -c is
mutually exclusive with the -p option (explicit list of packages) and
-n option (a number of packages, picked randomly).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, pkg-stats expects being executed from Buildroot's top-level
source directory. As we are going to extend pkg-stats to cover only
the packages available in the current configuration, it makes sense to
be able to run it from the output directory, which can be anywhere
compared to Buildroot's top-level directory.
This commit adjusts pkg-stats to this, by inferring all Buildroot
paths based on the location of the pkg-stats script itself.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When running the TestInitSystemSystemdRwIfupdown test, the rootfs must
be in read-write mode. The commit log [1] introducing systemd tests say
so:
"basic systemd, read-write, network w/ ifupdown"
With systemd 246.5, the service systemd-update-done return an error code
when it can't write on the filesystem (/etc)
[1] 117835d5fc
[2] 8019995e9a
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981813
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following flake8 warnings:
support/testing/tests/core/test_selinux.py:21:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:38:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:51:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:62:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:65:14: E127 continuation line over-indented for visual indent
support/testing/tests/init/test_systemd_selinux.py:53:1: E302 expected 2 blank lines, found 1
support/testing/tests/init/test_systemd_selinux.py:64:1: E302 expected 2 blank lines, found 1
Interestingly, the "continuation line over-indented for visual indent"
shows up only once, while the same pattern is there at multiple places
in the file. We fix all places with that over-indentation pattern.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Extract from bug report:
"Code line 120 to line 128 is to check whether the patch containing
"rename from" and "rename to". But it directly use grep to find,
ignoring the patch may be a tar file or else. It can only work on patch
of textfile form."
Fixes:
- https://bugs.buildroot.org/show_bug.cgi?id=11931
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 2020.08-1 release of Bootlin toolchains has brought support for 3
additional architecture variants, so let's support them.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
glibc toolchains must be disabled for static only configuration.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some externals may wish to provide custom init systems for tightly
integrated boot. This has been supported through the BR2_INIT_NONE,
however a downside to the BR2_INIT_NONE is it forces the custom init
system to use either skeleton-custom and roll a custom skeleton for
each target, or skeleton-init-none which isn't a complete skeleton.
Allowing br2-external to define custom BR2_INIT_* means they can now
safely 'select' the BR2_PACKAGE_SKELETON_INIT_*, and re-use any of the
skeletons in Buildroot, or one from a br2-external tree.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Today, the BR2_ROOTFS_SKELETON_CUSTOM is the only way to build a custom
skeleton. But it's limiting as users must provide a pre-built skeleton
for each target. Supporting a br2-external package allows users to build
up a skeleton and customize it with their own KConfig options.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
apply-patches currently blindly removes *.orig / .*.orig files as GNU patch
by default writes these as backup files when patches only apply with fuzz.
This is unfortunate as package sources may contain files ending in .orig as
well, breaking the build. Luckily GNU patch can be told to not write these
backup files using the --no-backup-if-mismatch option, so used that instead
of the .orig removal step.
--no-backup-if-mismatch is supported since GNU patch 2.3.8 (1997-06-17) and
busybox patch if built with CONFIG_DESKTOP, but E.G. isn't supported by the
BSD patch, so add logic to dependencies.sh to error out if patch doesn't
support the flag.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we handle three kinds of tests: basic, defconfig, and
runtime, and we treat them totally independently ones from the others.
Except for the basic tests that are ignored when defconfig or runtime
tests are explicitly requested.
The basic tests are also run systematically on all our reference
branches: master, next (when it exists), and the maintenance branches:
YYYY.MM.x.
Furthermore, we can see that the conditions to run each set of tests
are very similar, with only the explicit queries differing by name.
Rework the script so that the conditions are expressed only once, and
each set of tests is decided for each condition. This makes it easier
to decide what tests should run under what conditions.
Using GitLab-CI's schedules, with a variable expressing the actual test
to run, would seem the obvious choice to trigger the pipelines. However,
a schedule is configured for a specific branch, which means we would
need one schedule per branch we want to build per test cases we want to
run, *and* that we update those schedules when we add/remove branches
(e.g. when we open/close 'next', or a maintenance branch). This is not
very nice, as it requires some manual tweaking and twiddling on the web
UI.
Instead, we resort to using triggers, that will be triggered from a
cronjob on some server. Using a cronjiob allows us to more easily manage
the branches we want to test and test cases we want to run, to more
easily spread the load over the week, etc...
Note: triggering a pipeline can be done with a simple curl invocation:
$ curl -X POST \
-F "token=${YOUR_TOKEN}" \
-F "ref=${BRANCH_TO_TEST}" \
-F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \
"https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline"
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add tests to ensure the packages SELinux functionalities (being able to
select an extra SELinux module in the refpolicy, and being able to
provide a custom SELinux module) are working as expected.
We use a BR2_EXTERNAL folder, provided in the tests, to use a custom
SELinux enabled package.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for BR2_PACKAGE_REFPOLICY_CUSTOM_GIT (which allows to select
a custom location for the SELinux refpolicy). The test uses the official
refpolicy as a test (we only want to test the functionality is working,
not that another refpolicy is correctly building; that is an user
problematic).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for the BR2_REFPOLICY_EXTRA_MODULES_DIRS functionality (which
allows to provide custom SELinux modules).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch adds a test for the BR2_REFPOLICY_EXTRA_MODULES
functionality (which allows to select extra modules within the SELinux
refpolicy using Kconfig).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test called 'SELinuxSystemdSquashfs' which will perform the same
tests as the Ext4 version, but using a Squashfs filesystem. Thanks to
this, we'll have a test on a real only filesystem.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This adds a test called 'SELinuxSystemdExt4'. This test will build an
SELinux enabled image with systemd, boot it, and perform a few runtime
tests to check SELinux related capabilities.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 9e4ffdc8cf modified the output of
'setlocalversion' so that the Buildroot version tag is included in the
output, the version part was added in Makefile.
Due to differences in behavior of the used git and Mercurial commands, this
caused different output for the Mercurial case, in BR2_VERSION_FULL and thus
/etc/os-release and 'make print-version'. Assuming the official Buildroot
releases are tagged and no project-specific tags are present, the output
after commit 9e4ffdc8cf is:
-hg<commit>
whereas it is expected to be something like:
2020.02.6-hg<commit>
Change the Mercurial case in setlocalversion to behave similar to git,
looking up the latest tag if the current revision is not itself tagged.
The number of commits after the latest tag is not added, unlike in git, as
this value is not commonly present in Mercurial output, and its added value
can be disputed in this context. Even one commit could bring a huge change
to the sources, so in order to interpret the number one has to look at the
repository anyhow, in which case the commit ID can just be used.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a minimal s390x s13 autobuild configuration for the
internal toolchain with glibc.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Alexander Egorenkov <egorenar@linux.ibm.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit a2830f0dad (support/gnuconfig: bump version) carried
spurious, uncommited local changes to config.sub, that were not
part of upstream commit d7a4dee7cc25e332b990d0a6d9f0ddd42cb33cf5.
Fix that by actually using the code as it is upstream.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[yann.morin.1998@free.fr:
- squash the revert and the new bump into this commit
- ammend commit log accordingly
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When multiple conditions match simultaneously, even though that should
not happen in practice, we want the more "important" one to win over
the less "important" ones. For example, a tag is more important than a
branch name or a trigger.
Currently, the latest condition to match takes precendence over any
previous one, while we want the exact opposite.
Fix that with proper fallbacks in else-blocks.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes issues with the triple on IBM s390x and Z machines.
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: update both, using the update script]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add a new option that prints the (runtime) path of compiled .py files
when VERBOSE=1 is set.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.
By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.
This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.
Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.
Rework the script to call py_compile.compile() directly with pertinent
options:
- The script now has a new --strip-root argument. This argument is
optional but will always be specified when compiling py files in
buildroot.
- All other (non-optional) arguments are folders in which all
"importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
file is computed and encoded into the compiled .pyc.
No need to change directory before running the script anymore.
The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.
Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).
Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.
So we just drop support for --force, and instead we always byte-compile.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
- always byte-compile
- drop --force
- expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Only run code when the script is executed directly (not imported).
Factorize command description by using the script's __doc__ variable.
Fix typo in --force help message.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the check of defconfigs is run for all branches, even those
that are pushed only to run runtime tests. This is very inconvenient.
In fact, we only want to check the defconfigs on standard branches, that
is master, next, and the maintenance branches.
This will also decrease drastically the number gitlab-ci minutes used
when one pushes their repo to gitlab.com, where the number of CI minutes
are now going to be pretty severely restricted.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that those tests were so far ignored only when requesting a single
defconfig build, or a single runtime test build; everything else
was trigerring thoses tests.
However, it feels more natural that they are also ignored when all
defconfigs build. or all runtime tests, are explictly requested.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When we build the defconfigs, we already check they are correct, so
there is no need to run the correctness check explicitly.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Acked-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the image name and version are duplicated in the main
pipeline and the generated, child pipeline.
This is a condition for a future gaffe, so let's use the image from the
main pipeline when generating the child one.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This script is currently very crude, but we're going to extend it, at
which point it will be nicer to have functions, local variables, et al.
Introduce a main() in preparation of those future evolutions.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that one is silenced, rather than fixed: we indeed need to import
after we add the local directory to the modules search path.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
