apply-patches currently blindly removes *.orig / .*.orig files as GNU patch
by default writes these as backup files when patches only apply with fuzz.
This is unfortunate as package sources may contain files ending in .orig as
well, breaking the build. Luckily GNU patch can be told to not write these
backup files using the --no-backup-if-mismatch option, so used that instead
of the .orig removal step.
--no-backup-if-mismatch is supported since GNU patch 2.3.8 (1997-06-17) and
busybox patch if built with CONFIG_DESKTOP, but E.G. isn't supported by the
BSD patch, so add logic to dependencies.sh to error out if patch doesn't
support the flag.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we handle three kinds of tests: basic, defconfig, and
runtime, and we treat them totally independently ones from the others.
Except for the basic tests that are ignored when defconfig or runtime
tests are explicitly requested.
The basic tests are also run systematically on all our reference
branches: master, next (when it exists), and the maintenance branches:
YYYY.MM.x.
Furthermore, we can see that the conditions to run each set of tests
are very similar, with only the explicit queries differing by name.
Rework the script so that the conditions are expressed only once, and
each set of tests is decided for each condition. This makes it easier
to decide what tests should run under what conditions.
Using GitLab-CI's schedules, with a variable expressing the actual test
to run, would seem the obvious choice to trigger the pipelines. However,
a schedule is configured for a specific branch, which means we would
need one schedule per branch we want to build per test cases we want to
run, *and* that we update those schedules when we add/remove branches
(e.g. when we open/close 'next', or a maintenance branch). This is not
very nice, as it requires some manual tweaking and twiddling on the web
UI.
Instead, we resort to using triggers, that will be triggered from a
cronjob on some server. Using a cronjiob allows us to more easily manage
the branches we want to test and test cases we want to run, to more
easily spread the load over the week, etc...
Note: triggering a pipeline can be done with a simple curl invocation:
$ curl -X POST \
-F "token=${YOUR_TOKEN}" \
-F "ref=${BRANCH_TO_TEST}" \
-F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \
"https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline"
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add tests to ensure the packages SELinux functionalities (being able to
select an extra SELinux module in the refpolicy, and being able to
provide a custom SELinux module) are working as expected.
We use a BR2_EXTERNAL folder, provided in the tests, to use a custom
SELinux enabled package.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for BR2_PACKAGE_REFPOLICY_CUSTOM_GIT (which allows to select
a custom location for the SELinux refpolicy). The test uses the official
refpolicy as a test (we only want to test the functionality is working,
not that another refpolicy is correctly building; that is an user
problematic).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for the BR2_REFPOLICY_EXTRA_MODULES_DIRS functionality (which
allows to provide custom SELinux modules).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch adds a test for the BR2_REFPOLICY_EXTRA_MODULES
functionality (which allows to select extra modules within the SELinux
refpolicy using Kconfig).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test called 'SELinuxSystemdSquashfs' which will perform the same
tests as the Ext4 version, but using a Squashfs filesystem. Thanks to
this, we'll have a test on a real only filesystem.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This adds a test called 'SELinuxSystemdExt4'. This test will build an
SELinux enabled image with systemd, boot it, and perform a few runtime
tests to check SELinux related capabilities.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 9e4ffdc8cf modified the output of
'setlocalversion' so that the Buildroot version tag is included in the
output, the version part was added in Makefile.
Due to differences in behavior of the used git and Mercurial commands, this
caused different output for the Mercurial case, in BR2_VERSION_FULL and thus
/etc/os-release and 'make print-version'. Assuming the official Buildroot
releases are tagged and no project-specific tags are present, the output
after commit 9e4ffdc8cf is:
-hg<commit>
whereas it is expected to be something like:
2020.02.6-hg<commit>
Change the Mercurial case in setlocalversion to behave similar to git,
looking up the latest tag if the current revision is not itself tagged.
The number of commits after the latest tag is not added, unlike in git, as
this value is not commonly present in Mercurial output, and its added value
can be disputed in this context. Even one commit could bring a huge change
to the sources, so in order to interpret the number one has to look at the
repository anyhow, in which case the commit ID can just be used.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a minimal s390x s13 autobuild configuration for the
internal toolchain with glibc.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Alexander Egorenkov <egorenar@linux.ibm.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit a2830f0dad (support/gnuconfig: bump version) carried
spurious, uncommited local changes to config.sub, that were not
part of upstream commit d7a4dee7cc25e332b990d0a6d9f0ddd42cb33cf5.
Fix that by actually using the code as it is upstream.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[yann.morin.1998@free.fr:
- squash the revert and the new bump into this commit
- ammend commit log accordingly
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When multiple conditions match simultaneously, even though that should
not happen in practice, we want the more "important" one to win over
the less "important" ones. For example, a tag is more important than a
branch name or a trigger.
Currently, the latest condition to match takes precendence over any
previous one, while we want the exact opposite.
Fix that with proper fallbacks in else-blocks.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes issues with the triple on IBM s390x and Z machines.
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: update both, using the update script]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add a new option that prints the (runtime) path of compiled .py files
when VERBOSE=1 is set.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.
By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.
This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.
Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.
Rework the script to call py_compile.compile() directly with pertinent
options:
- The script now has a new --strip-root argument. This argument is
optional but will always be specified when compiling py files in
buildroot.
- All other (non-optional) arguments are folders in which all
"importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
file is computed and encoded into the compiled .pyc.
No need to change directory before running the script anymore.
The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.
Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).
Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.
So we just drop support for --force, and instead we always byte-compile.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
- always byte-compile
- drop --force
- expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Only run code when the script is executed directly (not imported).
Factorize command description by using the script's __doc__ variable.
Fix typo in --force help message.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the check of defconfigs is run for all branches, even those
that are pushed only to run runtime tests. This is very inconvenient.
In fact, we only want to check the defconfigs on standard branches, that
is master, next, and the maintenance branches.
This will also decrease drastically the number gitlab-ci minutes used
when one pushes their repo to gitlab.com, where the number of CI minutes
are now going to be pretty severely restricted.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that those tests were so far ignored only when requesting a single
defconfig build, or a single runtime test build; everything else
was trigerring thoses tests.
However, it feels more natural that they are also ignored when all
defconfigs build. or all runtime tests, are explictly requested.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When we build the defconfigs, we already check they are correct, so
there is no need to run the correctness check explicitly.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Acked-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the image name and version are duplicated in the main
pipeline and the generated, child pipeline.
This is a condition for a future gaffe, so let's use the image from the
main pipeline when generating the child one.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This script is currently very crude, but we're going to extend it, at
which point it will be nicer to have functions, local variables, et al.
Introduce a main() in preparation of those future evolutions.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Note that one is silenced, rather than fixed: we indeed need to import
after we add the local directory to the modules search path.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Move the code to run check-flake8 into the Makefile, like we have for
check-package, so that it is easy to run locally (and not wait for
someone to report a failure from their Gitlab pipelines).
Compared to the existing check from gitlab-ci.yml, the Makefile check
differs in this respect:
- don't explicitly find *.py files: they are supposed to also be found
as a result of running 'file' on them;
- use git ls-tree instead of find: this is supopsedly faster as it
uses the index rather than readdir();
- don't output the count of warnings or errors: the output is a single
integer, which is confusing when there are errors, and even more so
when there are no, when it is simply '0';
- don't sort: the output is already stable and independent from the
locale;
- don't report the number of processed files: this information is
rather useless, and getting a hold of it would be more challenging
in this new code.
Note: ideally, we would want to use --null, --zero, or similar options,
with utilities that generates or parses a files listing. While git
ls-tree and xargs do support it, it becomes a little bit tricky to use
the --print0 option of file, and then grep in that output (it is not
undoable, but would requires replacing grep+cut with some sed trickery).
Since we do not expect our scripts names to contain funky chars (like
\n or a colon), we just hand-wave away that issue (and the old code was
doing the same assumption too).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Current X.org X server is incompatible with this driver.
We no longer support unmaintainted versions of X.org X server.
Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When it was applied, commit 243d500f8d (support/testing: add openssh
runtime test) was amended to not provide a NIC to the emulated machine,
as the test did not require access to the outer world: it only uses the
lo interface. Also, there was a discrepancy between the NIC name in the
Buildroot configuration, and the drivers available in our default kernel
image, making the boot hang for a while whaiting for a NIC that would
never come.
However, that tweak was tested locally with a qmeu version more recent
than the one available in our buidroot/base Docker image. As a
consequence, that test fails to run in gitlab-ci.
Revert to using the old way of specifying no network: it works on
gitlab-ci, and qemu versions in standard distros still support it.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Drop the debug-level print as noticed by Titouan.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Titouan Christophe <titouan.christophe@railnova.eu>
This commit adds the new test cases generated automatically by the
bl-toolchains-gen script, to test the integration of the Bootlin
toolchains.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
https://toolchains.bootlin.com/ has been providing for a few years a
number of ready-to-use pre-built toolchains, for a wide range of
architectures (which it turns out, are all built using Buildroot).
While toolchains.bootlin.com provides Buildroot config fragments to
easily use those toolchains with Buildroot (see [0] for example), this
is not visible anywhere. So instead, we would like to add support for
these toolchains in Buildroot just like we have existing support for
Linaro, ARM, Synopsys, etc. toolchains.
[0] https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--bleeding-edge-2020.02-2.frag
However, the number of toolchains provided by toolchains.bootlin.com
is really large, and they are regularly updated. Maintaining that
manually would be time consuming and error-prone. So instead, this
commit introduces a script that automatically generates:
- toolchain/toolchain-external/toolchain-external-bootlin/Config.in.options
- toolchain/toolchain-external/toolchain-external-bootlin/toolchain-external-bootlin.mk
- toolchain/toolchain-external/toolchain-external-bootlin/toolchain-external-bootlin.hash
- support/testing/tests/toolchain/test_external_bootlin.py
We create a single external toolchain package, with a Kconfig "choice"
as a sub-option to select the toolchain variant to be used. The script
contains a Python dict that provides the mapping between the
toolchains provided by toolchains.bootlin.com, and the architecture
options/variants they are applicable to.
The test cases allow to verify that the toolchain configuration is
correct, and that it is able to build a Busybox based system. It
doesn't do any runtime testing as such testing is already done by
toolchains.bootlin.com: the test cases here are only meant to verify
that the toolchain-external-bootlin package works as expected.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This scripts takes as entry on stdin a JSON description of the package
used for a given configuration. This description is the one generated
by "make show-info".
The script generates the list of all the packages used and if they are
affected by a CVE. The output is either a JSON or an HTML file similar
to the one generated by pkg-stats.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>=
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The affects method of the CVE uses the Package class defined in
pkg-stats. The purpose of migrating the CVE class outside of pkg-stats
was to be able to reuse it from other scripts. So let's remove the
Package dependency and only use the needed information.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In 2019, the JSON vulnerability feeds switched their schema from
version 1.0 to 1.1.
The main difference is the removal of the "affects" element that we
were using to check if a package was affected by a CVE.
This information is now available in the "configuration" element which
contains the cpeid as well as properties about the versions
affected. Instead of having a list of the versions affected, with
these properties, it is possible to have a range of versions.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In order to be able to use the CVE checking logic outside of
pkg-stats, move the CVE class in a module that can be used by other
scripts.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some CVE entries in the NVD database have version_value set to "-",
which seems to indicate that it applies to all versions of the
software project, or that they don't really know which versions are
affected, and which are not.
So, for the benefit of doubt, it seems more appropriate to consider
such CVEs as affecting our packages.
This makes the total number of CVEs affecting our next branch jump
from 141 CVEs to 658 CVEs, but that number will go back down once we
switch to the JSON 1.1 schema. Indeed, in the JSON 1.0 schema, there
are often cases where a version_value is set to "=" *and* specific
versions are set to.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
As we recently stopped testing the x86-64 Sourcery toolchain, it means
we no longer have any x86-64 glibc based toolchain in our
autobuilders. Since this is a pretty common configuration, it makes
sense to test it, which this commit does by adding a config fragment
to use the x86-64 glibc bleeding edge Bootlin toolchain.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This toolchain uses an old gcc 6.2.0, and newer versions of the
toolchain are no longer publicly available. This old gcc 6.2.0 causes
build issues of Boost, which are unfixable without updating the
toolchain. As we're about to drop support for this toolchain entirely,
we must stop testing it in our autobuilder infrastructure.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Modeled after similar python packages.
However, this one is picky, and throws an exception when it
detects that it is not running on a Raspberry Pi. So we just
catch that exception and check this is what we expect.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Christian Stewart <christian@paral.in>
Cc: Michael Fischer <mf@go-sys.de>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Ian Haylock <haylocki@yahoo.co.uk>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This new runtime test is based on test_dropbear.py. The only required change
is to use "-oStrictHostKeyChecking=no" instead of "-y" to accept the new key.
Since the base test infra only provide a uClibc-ng toolchain, add a second
test using a glibc based internal toolchain.
For example, this allow to trigger the openssh 8.1p bug with glibc 2.31 [1].
[1] https://bugs.archlinux.org/task/65386
Signed-off-by: Romain Naour <romain.naour@smile.fr>
yann.morin.1998@free.fr:
- deduplicate the whole test
- don't provide any NIC, we only need and use lo
- simplify post-build script (append with cat, don't munge with sed)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The TestExternalToolchain() base class implement a test checking if
the ELF interpreter that is advertised by Busybox really exists in the
rootfs. Of course, this only makes sense with ELF toolchains. Until
now, only ELF toolchains were tested, but we are going to use
TestExternalToolchain() with non-ELF toolchains as well, so let's make
this conditional.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: strip() lines during readlines()]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
