Commit 22b6945552 (support/scripts/cve.py: switch from NVD to FKIE for
the JSON files) had to change the decompressor from gz to xz, as the new
location is using xz compression.
That commit mentioned that it was spawning an external xz process to do
the decompression, on the pretence that "there is no xz decompressor in
Python stdlib."
Before version 3.1, ijson.items() only accepted a file-like object as
input (that file-like object could yield bytes() or str(), both were
supported). Starting with version 3.1, ijson.items() also accepts that
it be directly passed bytes() or str() directly. subprocess.check_output()
means we are now passing bytes() to ijson.items(), so it fails on ijson
versions before 3.1, with failures such as:
[...]
File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer
if type(f.read(0)) == bytetype:
AttributeError: 'bytes' object has no attribute 'read'
Ubuntu 20.04, on which the pkg-stats run to generate the daily report,
only has ijson 2.3. More recent distros have more recent versions of
ijson, like Fedora 39 that has 3.2.3, recent enough to support being fed
bytes(). Commit 22b6945552 was tested on Fedora 39, so did not catch
the issue.
However, the reasoning in 22b6945552 is wrong: there *is* the lzma
module, at least since python 3.3 (that is, aeons ago), which is able to
read xz-compressed files; it also has an API similar to the gzip module,
and can provide a file-like object that exposes the decompressed data.
So, do just that: provide an lzma-wrapped file-like object to ijson, so
that we can eventually recover our daily reports that everything is
broken! :-]
Note that this construct still works on recent versions!
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit f71d9f49e5 (support/scripts/pkg-stats: fix datetime deprecation
warning) forgot to consider that the datetime.UTC suggested by python
3.12, was only introduced with python 3.11.
However, we are still generating the daily report on a python 3.8
version, which fails at runtime:
AttributeError: module 'datetime' has no attribute 'UTC'
It turns out that datetime.UTC is just an alias for datetime.timezone.utc,
which seems to have existed since before python3...
Use datetime.timezone.utc instead of its alias.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
While the old NVD JSON feed provided data files where the CVEs were
sorted by ID, the new feed from FKIE does not have sorted CVEs.
Add a method to sort a list of CVE IDs (i.e. CVE ID strings, not CVE
objects!), and use that when emiting the HTML output.
The JSON output need not be sorted, because it is supposed to be used
for post-processing, and we do not care about the ordering there; a
consumer interested in sorting should sort on their side.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Commit 22b6945552 (support/scripts/cve.py: switch from NVD to FKIE for
the JSON files) missed the fact that the layout of the FKIE data files
are different from the original NVD ones. They are formatted according
to the NVD v2 API.
Most differences are relatively trivial fields renaming, and those are
easily spotted in this patch.
There is however one key difference in the layout of the configurations.
Where the NVD had "configurations" as an object with a "nodes" key, the
FKIE has a "configurations" as a list of objects with a single "nodes"
key; i.e. it is one-level deeper.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Abide by the warning reported with python 3.12:
.../support/scripts/pkg-stats:1289: DeprecationWarning:
datetime.datetime.utcnow() is deprecated and scheduled for removal
in a future version. Use timezone-aware objects to represent
datetimes in UTC: datetime.datetime.now(datetime.UTC).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Currently, when the version string is "too long", it is arbitrarily
truncated.
This works well for commit hashes, because usually the truncation is
long enough to provide a short hash that is still unique in the
upstream VCS.
However, there are non-hash-like versions strings that get truncated
and wihch the discriminant part is toward the end.
Yet, adapting the version cell to the widest versions string (most
probably a git hash) is not very interesting; the table is already very
large.
Make the cell with the version string scrollable: we get to keep the
best of both worlds: a narrow version cell, and a full-length version
string that can be copy-pasted if needed.
Signed-off-by: Sen Hastings <sen@hastings.org>
[yann.morin.1998@free.fr: reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit dfed5acb56 ("utils/check-package: use https for the manual URL")
replaced the default url to the Buildroot manual while it was used
by TestCheckPackage test.
Update TestCheckPackage with https url.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6224243484
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of waiting for a hardcoded time of 30s we check periodically every
second if the server is already up. If it isn't up after the full timeout
(which is the same as before) expired the test fails.
We need to redirect all output of the background started task to
/dev/null now as it otherwise confuses the emulator.run() exit code
parsing logic (as it gets out of order messages from the emulator).
Signed-off-by: Marcus Hoffmann <buildroot@bubu1.eu>
yann.morin.1998@free.fr: simplify assert test]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Similar to the new fastapi test, instead of waiting for a hard coded
amount of time we can retry every second until the server is available
and abort if after the timeout we still didn't manage to connect.
Signed-off-by: Marcus Hoffmann <buildroot@bubu1.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
gpg key generation can take longer than the default timeout on a
loaded or slow test host. The commit increase the timeout for the
key generation command to prevent the test to randomly fail.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The symmetric encryption test can sometimes take longer than the default
timeout. This commit increase the timeout to 10 seconds for that
command.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When the CVE lookup was added in commit
4a157be9ef, the starting year of the JSON
files was set to 2002. However, there are also CVEs from 1999, 2000 and
2001. It is not clear why these were skipped back then.
Set the start year to 1999 to capture these old CVEs too.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
NVD will deprecate the v1.1 API which allows us to download the full
database as individual JSON files. Instead, there's a horribly crappy
API that is extremely slow and subject to race conditions.
Fortunately, there is a project, Fraunhofer FKIE - Cyber Analysis and
Defense [1], that goes through the effort of adapting to this new API
and regenerating the convenient JSON files. The JSON files and meta
files are re-generated daily.
Instead of implementing the NVD v2 API, we decided to just use the JSON
files generatd by fkie-cad. That saves us the effort of solving the race
conditions, devising a cache mechanism that works, handling the frequent
gateway timeouts on the NVD servers, dealing with the rate limiting, and
keeping up with changes in the API.
Switch to this repository on github as NVD_BASE_URL. The file name is
also slightly different (CVE-20XX.json instead of nvdcve-1.1-20XX.json).
The fkie-cad repository compresses with xz instead of gz. Therefore:
- rename the filename variables to _xz instead of _gz;
- use xz as a subprocess because there is no xz decompressor in Python
stdlib.
[1] https://www.fkie.fraunhofer.de/en/departments/cad.html
Cc: Daniel Lang <dalang@gmx.at>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We currently check the CPE database to see if the CPE ID we use
(including the version) is already in the database.
However, the version part of the CPE ID is not actually used for CVE
matching. Instead, the CVEs have a range of versions associated with
them and we match against those ranges.
In addition, NVD is moving to a new API for accessing the CPE database.
It will not longer be possible to simply download all the CPE IDs, and
due to rate limiting, the download will have to be done in several
queries.
Since all of this is anyway of limited use, drop the CPE database lookup
entirely. Instead, as long as a CPE ID is defined in a package, it is
considered OK, without any checks.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The meson upgrade from 1.1.1 to 1.2.0 have been reverted because
host-qemu (version v8.0.3) was searching libraries in the build machine
instead of buildroot host directory [1].
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-L/usr/lib/gcc/x86_64-linux-gnu/10 \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib -L/usr/lib \
-L/lib/x86_64-linux-gnu \
-L/lib/../lib \
-L/usr/lib/../lib \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../.. \
-L/lib \
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Upgrading qemu to version v8.1.0 solves the problem:
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Those two tests were done with the same buildroot version (2023.08) on
the same machine were the bug appeared. Using, git bisect, in this
environment, it was found that the problem was fixed in qemu by the
commit [2].
Thus, it is possible that the problem was introduced by improper use of
meson by qemu <= v8.0.3. Also, meson has been updated to version 1.2.1
and 1.3.0 in yocto and it doesn't seem to create any issue. [3]
The original problem is no longer present and some projects require
meson > 1.1.1 (e.g. rusticl requires meson 1.2.0 since mesa 23.3.0 and
meson 1.3.1 since mesa 24.0.0).
"pkgconfig" field is deprecated by meson 1.3.0 and replaced by
"pkg-config". See: [4]
The patch that adds the pkg_config_static property has been rebased.
[1] acfdf21f0b
[2] https://gitlab.com/qemu-project/qemu/-/commit/Fc9a809e0d28417fa1e7f7efc19f845bda4c1be9
[3] https://github.com/yoctoproject/poky/commits/master/meta/recipes-devtools/meson
[4] https://mesonbuild.com/Release-notes-for-1-3-0.html#machine-files-pkgconfig-field-deprecated-and-replaced-by-pkgconfig
Signed-off-by: Antoine Coutant <antoine.coutant@smile.fr>
[yann.morin.1998@free.fr:
- keep our formatting in the patch
- reflow commit log (to keep busy while test-building!)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
TestGlxinfo fail with a new runtime error:
# glxinfo -B -display :0
name of display: :0
traps: glxinfo[84] trap invalid opcode ip:b73c7027 sp:bf8433c0 error:0 in swrast_dri.so[b6e4c000+64f000]
Illegal instruction
The x86-core2 Bootlin toolchains are built for a core2 CPU [0],
this means that the Bootlin toolchains may use core2-specific
instructions.
The TestGlxinfo test is setup for BR2_x86_core2, so our
executables will also contain core2 instructions.
However, the default Qemu x86 is not guaranteed to emulate all the
instructions specific to core2, causing runtime issues as reported
above.
A similar issue has been fixed by adding Nehalem cpu emulation on
the qemu command line. See 4f565b5222 ("support/testing: use Nehalem
cpu emulation for TestGrubX8664EFI").
Set core2duo cpu emulation for TestGlxinfo on the qemu command line.
[0] https://gitlab.com/buildroot.org/toolchains-builder/-/blob/kubu/toolchain-builder-2023.08/configs/arch/x86-core2.config?ref_type=heads
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The current package has not recieved an update since Sat Oct 9 2021
33ece2446e and is not python 3.12 compatible.
Furthermore, the current version requires at least 42 new packages worth of
depedencies of which several require patches to be python 3.12 compatible.
As nobody has stepped up to maintain the package and its ever-growing list of
dependencies, along with the other problems, it is time to drop the package.
Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Search the external trees for package files and add them to the list.
The list of directories walked and excluded are the same as for the main
tree, and should work out of the box if the user sticks to the directory
structure suggested in the manual.
Two additional properties were added to the Package class, the tree name and
the path. For consistency and to simplify the code, packages in the main tree
are marked as coming from "BR2".
The HTML output has a new column listing the external name (or "BR2") and the
json output has a new property "tree".
Signed-off-by: Juan Carrano <juan.carrano@ebee.berlin>
[Arnout:
- fix flake8 error "'itertools' imported but unused";
- use str.split instead of str.partition;
- use BR2_EXTERNAL_BUILDROOT_PATH instead of BR2_EXTERNAL_BR2_PATH;
- remove pkgdir variable, instead use self.pkgdir.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Add a runtime test for fastapi. Use uvicorn as the asgi server
application as does the fastapi hello world example [1].
Fastapi depends on PydanticV2 now which is written in rust so we need to
run the test on armv7.
[1] https://fastapi.tiangolo.com/tutorial/first-steps/
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
[Arnout:
- fix flake8 errors
support/testing/tests/package/sample_python_fastapi.py:5:1: E302 expected 2 blank lines, found 1
support/testing/tests/package/sample_python_fastapi.py:8:1: W391 blank line at end of file
- Remove BR2_CCACHE (as requested by Marcus).
- Add a comment explaining that this also tests uvicorn and pydantic.
- Re-try wget in a loop instead of a fixed timeout of 30 seconds.
- Add a DEVELOPERS entry.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Under some conditions (mostly slow execution due to test host load),
the netcat runtime test can randomly fail. This is due to several
facts:
- the sleep time between the server and client is too short,
- the use of netcat option -c could close the connection before the
server could receive all the data.
This commit improves the test robustness by increasing the sleep time,
and by reducing the amount of transferred data (from 1MB of random data,
to a simpler string of few bytes). Also, to make sure netcat cannot wait
on DNS resolution, this commit also adds the -n option.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6093854664
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
In commit 7e0e6e3b86 (toolchain/toolchain-external/toolchain-external-bootlin:
update to 2023.11-1) the TestGrubX8664EFI fails with this error:
# efivar -l
traps: efivar[86] trap invalid opcode ip:7fc187f4c7f4 sp:7fff9bbaa930 error:0 in libefivar.so.1.38[7fc187f4c000+16000]
Illegal instruction
This error can be reproduced by installing other packages like "file".
The 2023.11-1 Bootlin toolchains are built for a corei7 CPU [0], which
is in fact a Nehalem CPU; we switched to the new names in commit
653fa001f3 (arch/Config.in.x86: add "newer" names for several Intel
x86 CPU variants). This means that the Bootlin toolchains may use
Nehalem-specific instructions.
The TestGrubX8664EFI test is also setup for BR2_x86_corei7, so our
executables will also contain Nehalem instructions.
However, the default Qemu x86_64 is not guaranteed to emulate all the
instructions specific to Nehalem, causing runtime issues as reported
above.
A similar issue has been fixed in toolchain-builder by adding Nehalem
cpu emulation on the qemu command line [0].
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/6093853712
[0] f2b253732b
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[yann.morin.1998@free.fr:
- reword the commit log to explain corei7 -> Nehalem equivalence
- note that the toolchain-builder *and* our test target corei7, thus
Nehalem
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
[yann.morin.1998@free.fr: add the comment to explain failure is success]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 8f403f0 "package/micropython-lib: merge with, and install
as part of micropython" brought micropython library within the
package.
This commit improves the micropython runtime test by enabling the
micropython-lib and by also adding a runtime test using one of its
module. We choose to use the gzip module, and check whther decodign q
simple gziped file works; the micropython-lib gzip can only decompress,
so we need to prepare it from the shell.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The rootfs generated TestPythonPy3SciPy by doesn't fit anymore in 120M.
Increase the rootfs size to 250M and checking the amount of space
available on the file system.
Filesystem Size Used Available Use% Mounted on
/dev/root 232.3M 134.5M 85.3M 61% /
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/5880448601
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Reviewed-by: Antoine Coutant <antoine.coutant@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
