support/scripts/pkg-stats:81:22: E211 whitespace before '('
support/scripts/pkg-stats:404:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/pkg-stats:561:12: E713 test for membership should be 'not in'
support/scripts/pkg-stats:567:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:595:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1051:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1057:1: E302 expected 2 blank lines, found 1
Also fix:
support/scripts/pkg-stats:1054:5: E722 do not use bare 'except'
found by a more recent flake8 version. The exception may be either
IndexError or AttributeError, so use Exception to catch either.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This commit improves pkg-stats to fill in pkg.status['cve'] depending
on the situation for CVEs affecting this package. They are then used
in the HTML rendering.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Virtual packages (with in pkg-stats speak have "no valid
infrastructure") and packages that have no version specified cannot be
used for CVE checking. They trigger a bunch of warnings from the CVE
checking code, as it cannot parse their version: they don't have any
version. So instead, we simply skip those packages.
A follow-up commit will improve the reporting to be able to
distinguish those packages from packages that have seen their CVEs
checked and don't have any reported.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit modifies cve.py, as well as its users cve-checker and
pkg-stats to support CPE ID based matching, for packages that have CPE
ID information.
One of the non-trivial thing is that we can't simply iterate over all
CVEs, and then iterate over all our packages to see which packages
have CPE ID information that match the CPEs affected by the
CVE. Indeed, this is an O(n^2) operation.
So instead, we do a pre-filtering of packages potentially affected. In
check_package_cves(), we build a cpe_product_pkgs dict that associates
a CPE product name to the packages that have this CPE product
name. The CPE product name is either derived from the CPE information
provided by the package if available, and otherwise we use the package
name, which is what was used prior to this patch.
And then, when we look at CVEs, we only consider the packages that
have a CPE product name matching the CPE products affected by the
CVEs. This is done in check_package_cve_affects().
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit improves the pkg-stats script to show the CPE ID of
packages, if available. For now, it doesn't use CPE IDs to match CVEs.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The logic in gen-bootlin-toolchains was assuming all glibc toolchains
have RPC support, which is no longer true since glibc 2.32 has dropped
RPC support.
It turns out that gen-bootlin-toolchains already had some proper logic
that selects BR2_TOOLCHAIN_HAS_NATIVE_RPC depending on the presence of
BR2_TOOLCHAIN_EXTERNAL_INET_RPC in the toolchain fragment. As such
toolchain fragments have been fixed in https://toolchains.bootlin.com,
we can now rely on this to properly decide if the toolchain has RPC
support or not.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 804a9e1865)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
quazip requires cmake 3.15 since version 1.0 and
89e7c201f0818adc8224
The rationale for this requirement is that "default locations for the
install(TARGETS command based on the GNUInstallDirs package were only
added in 3.14" and "3.15 is not that much of a difference from 3.14 and
it introduced a lot of useful UI improvements.":
https://github.com/stachenov/quazip/issues/82
Fixes:
- http://autobuild.buildroot.org/results/5d848a46109aef448ea1d1b857a500d9461dc2d9
Note: we also have some patches to allow some packages to build with
cmake-3.10, and this will not be tenable over the long run.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[yann.morin.1998@free.fr: add the "note"]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When boot-qemu-image.py script was added, we wanted to run
each qemu defconfig in gitlab, so we expect that all qemu
defconfig generate the script start-qemu.sh in images
directory.
Don't make it a hard requirement even if we prefer to be
able to do a runtime test for each qemu defconfig.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds a number of test cases to verify that the CPE_ID_*
variables are properly handled by the generic package infrastructure
and that the "make show-info" JSON output matches what we expect.
A total of 5 different example packages are used to exercise different
scenarios of CPE_ID_* variables usage.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the version encoded in a CPE is '-', we assume all
versions are affected, but when it's '*' with no further range
information, we assume no version is affected.
This doesn't make sense, so instead, we handle '*' and '-' in the same
way. If there's no version information available in the CVE CPE ID, we
assume all versions are affected.
This increases quite a bit the number of CVEs and package affected:
- "total-cves": 302,
- "pkg-cves": 100,
+ "total-cves": 597,
+ "pkg-cves": 135,
For example, CVE-2007-4476 has a CPE ID of:
cpe:2.3🅰️gnu:tar:*:*:*:*:*:*:*:*
So it should be taken into account. In this specific case, it is
combined with an AND with CPE ID
cpe:2.3⭕suse:suse_linux:10:*:enterprise_server:*:*:*:*:* but since
we don't support this kind of matching, we'd better be on the safe
side, and report this CVE as affecting tar, do an analysis of the CVE
impact, and document it in TAR_IGNORE_CVES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We should not rely on host installed bison/flex for target code. This
ensures better reproducibility of generated code.
http://lists.busybox.net/pipermail/buildroot/2020-November/296786.html
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that pkg-stats is able to generate its output based on the list of
packages enabled in the current configuration, cve-checker doesn't
serve any purpose.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
pkg-stats was initially a Buildroot maintenance oriented tool: it was
designed to examine all Buildroot packages and provide
statistics/details about them.
However, it turns out that a number of details provided by pkg-stats,
especially CVEs, are relevant also for Buildroot users, who would like
to check regularly if their specific Buildroot configuration is
affected by CVEs or not, and possibly check if all packages have
license information, license files, etc.
The cve-checker script was recently introduced to provide an output
relatively similar to pkg-stats, but focused on CVEs only.
But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses
"make show-info" to only consider packages enabled in the current
configuration.
So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
to generate its output based on the list of configured packages. -c is
mutually exclusive with the -p option (explicit list of packages) and
-n option (a number of packages, picked randomly).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, pkg-stats expects being executed from Buildroot's top-level
source directory. As we are going to extend pkg-stats to cover only
the packages available in the current configuration, it makes sense to
be able to run it from the output directory, which can be anywhere
compared to Buildroot's top-level directory.
This commit adjusts pkg-stats to this, by inferring all Buildroot
paths based on the location of the pkg-stats script itself.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When running the TestInitSystemSystemdRwIfupdown test, the rootfs must
be in read-write mode. The commit log [1] introducing systemd tests say
so:
"basic systemd, read-write, network w/ ifupdown"
With systemd 246.5, the service systemd-update-done return an error code
when it can't write on the filesystem (/etc)
[1] 117835d5fc
[2] 8019995e9a
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981813
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following flake8 warnings:
support/testing/tests/core/test_selinux.py:21:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:38:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:51:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:62:1: E302 expected 2 blank lines, found 1
support/testing/tests/core/test_selinux.py:65:14: E127 continuation line over-indented for visual indent
support/testing/tests/init/test_systemd_selinux.py:53:1: E302 expected 2 blank lines, found 1
support/testing/tests/init/test_systemd_selinux.py:64:1: E302 expected 2 blank lines, found 1
Interestingly, the "continuation line over-indented for visual indent"
shows up only once, while the same pattern is there at multiple places
in the file. We fix all places with that over-indentation pattern.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Extract from bug report:
"Code line 120 to line 128 is to check whether the patch containing
"rename from" and "rename to". But it directly use grep to find,
ignoring the patch may be a tar file or else. It can only work on patch
of textfile form."
Fixes:
- https://bugs.buildroot.org/show_bug.cgi?id=11931
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 2020.08-1 release of Bootlin toolchains has brought support for 3
additional architecture variants, so let's support them.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
glibc toolchains must be disabled for static only configuration.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some externals may wish to provide custom init systems for tightly
integrated boot. This has been supported through the BR2_INIT_NONE,
however a downside to the BR2_INIT_NONE is it forces the custom init
system to use either skeleton-custom and roll a custom skeleton for
each target, or skeleton-init-none which isn't a complete skeleton.
Allowing br2-external to define custom BR2_INIT_* means they can now
safely 'select' the BR2_PACKAGE_SKELETON_INIT_*, and re-use any of the
skeletons in Buildroot, or one from a br2-external tree.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Today, the BR2_ROOTFS_SKELETON_CUSTOM is the only way to build a custom
skeleton. But it's limiting as users must provide a pre-built skeleton
for each target. Supporting a br2-external package allows users to build
up a skeleton and customize it with their own KConfig options.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
apply-patches currently blindly removes *.orig / .*.orig files as GNU patch
by default writes these as backup files when patches only apply with fuzz.
This is unfortunate as package sources may contain files ending in .orig as
well, breaking the build. Luckily GNU patch can be told to not write these
backup files using the --no-backup-if-mismatch option, so used that instead
of the .orig removal step.
--no-backup-if-mismatch is supported since GNU patch 2.3.8 (1997-06-17) and
busybox patch if built with CONFIG_DESKTOP, but E.G. isn't supported by the
BSD patch, so add logic to dependencies.sh to error out if patch doesn't
support the flag.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, we handle three kinds of tests: basic, defconfig, and
runtime, and we treat them totally independently ones from the others.
Except for the basic tests that are ignored when defconfig or runtime
tests are explicitly requested.
The basic tests are also run systematically on all our reference
branches: master, next (when it exists), and the maintenance branches:
YYYY.MM.x.
Furthermore, we can see that the conditions to run each set of tests
are very similar, with only the explicit queries differing by name.
Rework the script so that the conditions are expressed only once, and
each set of tests is decided for each condition. This makes it easier
to decide what tests should run under what conditions.
Using GitLab-CI's schedules, with a variable expressing the actual test
to run, would seem the obvious choice to trigger the pipelines. However,
a schedule is configured for a specific branch, which means we would
need one schedule per branch we want to build per test cases we want to
run, *and* that we update those schedules when we add/remove branches
(e.g. when we open/close 'next', or a maintenance branch). This is not
very nice, as it requires some manual tweaking and twiddling on the web
UI.
Instead, we resort to using triggers, that will be triggered from a
cronjob on some server. Using a cronjiob allows us to more easily manage
the branches we want to test and test cases we want to run, to more
easily spread the load over the week, etc...
Note: triggering a pipeline can be done with a simple curl invocation:
$ curl -X POST \
-F "token=${YOUR_TOKEN}" \
-F "ref=${BRANCH_TO_TEST}" \
-F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \
"https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline"
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add tests to ensure the packages SELinux functionalities (being able to
select an extra SELinux module in the refpolicy, and being able to
provide a custom SELinux module) are working as expected.
We use a BR2_EXTERNAL folder, provided in the tests, to use a custom
SELinux enabled package.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for BR2_PACKAGE_REFPOLICY_CUSTOM_GIT (which allows to select
a custom location for the SELinux refpolicy). The test uses the official
refpolicy as a test (we only want to test the functionality is working,
not that another refpolicy is correctly building; that is an user
problematic).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test for the BR2_REFPOLICY_EXTRA_MODULES_DIRS functionality (which
allows to provide custom SELinux modules).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch adds a test for the BR2_REFPOLICY_EXTRA_MODULES
functionality (which allows to select extra modules within the SELinux
refpolicy using Kconfig).
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a test called 'SELinuxSystemdSquashfs' which will perform the same
tests as the Ext4 version, but using a Squashfs filesystem. Thanks to
this, we'll have a test on a real only filesystem.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This adds a test called 'SELinuxSystemdExt4'. This test will build an
SELinux enabled image with systemd, boot it, and perform a few runtime
tests to check SELinux related capabilities.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 9e4ffdc8cf modified the output of
'setlocalversion' so that the Buildroot version tag is included in the
output, the version part was added in Makefile.
Due to differences in behavior of the used git and Mercurial commands, this
caused different output for the Mercurial case, in BR2_VERSION_FULL and thus
/etc/os-release and 'make print-version'. Assuming the official Buildroot
releases are tagged and no project-specific tags are present, the output
after commit 9e4ffdc8cf is:
-hg<commit>
whereas it is expected to be something like:
2020.02.6-hg<commit>
Change the Mercurial case in setlocalversion to behave similar to git,
looking up the latest tag if the current revision is not itself tagged.
The number of commits after the latest tag is not added, unlike in git, as
this value is not commonly present in Mercurial output, and its added value
can be disputed in this context. Even one commit could bring a huge change
to the sources, so in order to interpret the number one has to look at the
repository anyhow, in which case the commit ID can just be used.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a minimal s390x s13 autobuild configuration for the
internal toolchain with glibc.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Alexander Egorenkov <egorenar@linux.ibm.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit a2830f0dad (support/gnuconfig: bump version) carried
spurious, uncommited local changes to config.sub, that were not
part of upstream commit d7a4dee7cc25e332b990d0a6d9f0ddd42cb33cf5.
Fix that by actually using the code as it is upstream.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
[yann.morin.1998@free.fr:
- squash the revert and the new bump into this commit
- ammend commit log accordingly
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When multiple conditions match simultaneously, even though that should
not happen in practice, we want the more "important" one to win over
the less "important" ones. For example, a tag is more important than a
branch name or a trigger.
Currently, the latest condition to match takes precendence over any
previous one, while we want the exact opposite.
Fix that with proper fallbacks in else-blocks.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes issues with the triple on IBM s390x and Z machines.
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
[yann.morin.1998@free.fr: update both, using the update script]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add a new option that prints the (runtime) path of compiled .py files
when VERBOSE=1 is set.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.
By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.
This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.
Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.
Rework the script to call py_compile.compile() directly with pertinent
options:
- The script now has a new --strip-root argument. This argument is
optional but will always be specified when compiling py files in
buildroot.
- All other (non-optional) arguments are folders in which all
"importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
file is computed and encoded into the compiled .pyc.
No need to change directory before running the script anymore.
The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.
Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).
Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.
So we just drop support for --force, and instead we always byte-compile.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
- always byte-compile
- drop --force
- expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
