For the new patch, adding block size options (commit 555f8dfd),
Yann E. MORIN requested updated testcases that specifically ensure
the extreme blocksizes (4K and 1024K) don't cause issues.
This patch splits the current test case in 2, testing with both
block sizes and ensuring the block size was applied in the same
fashion as for the specified compression.
Signed-off-by: Linus Kaschulla <linus@cosmos-ink.net>
[yann.morin.1998@free.fr: keep exisitng test with default size]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
While building the kernel tools, libelf header is missing:
output/TestZfsGlibc/build/linux-5.15.35/tools/objtool/include/objtool/elf.h:10:10: fatal error: gelf.h: No such file or directory
10 | #include <gelf.h>
Select BR2_LINUX_KERNEL_NEEDS_HOST_LIBELF to build host-libelf.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2429014008
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The commit [1] introcuded TestZfsBase as a common function
between all Zfs tests. But TestZfsBase test is executed
as a test itself.
Rename test_run() to base_test_run() to avoid this issue.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2429014006
[1] 593e8cb71f
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As reported by [1], the lxc test is broken since lxc >= 4.0.11.
A patch was added to lxc 4.0.11 to use the new mount api for devpts
setup [2] but the fall back code doesn't work when this new mount
API is not supported. This API was added in kernel 5.6.
(kernel 5.5)
DEBUG conf - conf.c:lxc_setup_devpts_child:1682 - No new devpts instance will be
mounted since no pts devices are required
lxc-start lxc_iperf3 DEBUG conf - conf.c:lxc_setup_dev_console:1966 - Cleared
all (0) mounts from "/dev/console"
lxc-start lxc_iperf3 ERROR mount_utils - mount_utils.c:mount_at:661 - No such
file or directory - Failed to mount "/proc/self/fd/44" to "/proc/self/fd/43"
lxc-start lxc_iperf3 ERROR conf - conf.c:lxc_setup_dev_console:1988 - No such
file or directory - Failed to mount "10(/dev/pts/0)" on "43"
lxc-start lxc_iperf3 ERROR conf - conf.c:lxc_setup_console:2143 - No such file
or directory - Failed to setup console
(kernel 5.6)
lxc-start lxc_iperf3 TRACE mount_utils - mount_utils.c:can_use_mount_api:582 -
Kernel supports mount api
lxc-start lxc_iperf3 TRACE mount_utils - mount_utils.c:move_detached_mount:328
- Attach detached mount 45 to filesystem at 43
lxc-start lxc_iperf3 TRACE conf - conf.c:lxc_setup_dev_console:1990 - Setup
console "/dev/pts/0"
Bump the kernel to the current LTS 5.15.38 version that fully support the
mount API needed by lxc.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2429013708
[1] http://lists.busybox.net/pipermail/buildroot/2022-January/635251.html
[2] be606e16fd
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add TestZfsBase that contains the common parts of the test.
Signed-off-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
GNU Octave is a high-level language, primarily intended for numerical
computations. It provides a convenient command line interface for
solving linear and nonlinear problems numerically, and for performing
other numerical experiments using a language that is mostly compatible
with Matlab. It may also be used as a batch-oriented language. Octave
has extensive tools for solving common numerical linear algebra
problems, finding the roots of nonlinear equations, integrating
ordinary functions, manipulating polynomials, and integrating ordinary
differential and differential-algebraic equations. It is easily
extensible and customizable via user-defined functions written in
Octave's own language, or using dynamically loaded modules written in
C++, C, Fortran, or other languages.
https://www.octave.org/
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Removed 0001-Correct-a-flaw-in-the-Python-3-version-checking.patch
because is already merged.
Select libcurl as required dependency, because keylocation now
supports https. OpenSSL was already a dependency, so libcurl will be
built with https support.
Add upstream patch to support uClibc.
We update the test cases to use the latest LTS kernel, 5.15.x.
Signed-off-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
pkg-stats currently uses the services from support/scripts/cpedb.py to
match the CPE identifiers of packages with the official CPE database.
Unfortunately, the cpedb.py code uses regular ElementTree parsing,
which involves loading the full XML tree into memory. This causes the
pkg-stats process to consume a huge amount of memory:
thomas 1310458 85.2 21.4 3708952 3450164 pts/5 R+ 16:04 0:33 | | \_ python3 ./support/scripts/pkg-stats
So, 3.7 GB of VSZ and 3.4 GB of RSS are used by the pkg-stats
process. This is causing the OOM killer to kick-in on machines with
relatively low memory.
This commit reimplements the XML parsing needed to do the CPE matching
directly in pkg-stats, using the XmlParser functionality of
ElementTree, also called "streaming parsing". Thanks to this, we never
load the entire XML tree in RAM, but only stream it through the
parser, and construct a very simple list of all CPE identifiers. The
max memory consumption of pkg-stats is now:
thomas 1317511 74.2 0.9 381104 152224 pts/5 R+ 16:08 0:17 | | \_ python3 ./support/scripts/pkg-stats
So, 381 MB of VSZ and 152 MB of RSS, which is obviously much better.
The JSON output of pkg-stats for the full package set, before and after
this commit, is exactly identical.
Now, one will probably wonder why this isn't directly changed in
cpedb.py. The reason is simple: cpedb.py is also used by
support/scripts/missing-cpe, which (for now) heavily relies on having
in memory the ElementTree objects, to re-generate a snippet of XML
that allows us to submit to NIST new CPE entries.
So, future work could include one of those two options:
(1) Re-integrate cpedb.py into missing-cpe directly, and live with
two different ways of processing the CPE database.
(2) Rewrite the missing-cpe logic to also be compatible with a
streaming parsing, which would allow this logic to be again
shared between pkg-stats and missing-cpe.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- add missing import of requests
- import CPEDB_URL from cpedb, instead of duplicating it
- fix flake8 errors
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Some upstream sites are very slow to respond, and the default timeout
of 300 seconds of the aiohttp.ClientSession() is too long. Let's
reduce it to 15 seconds.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This is useful when debugging/developing the pkg-stats script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This is useful when debugging/developing the pkg-stats script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This commit adds support for a new type of graph, showing the timeline
of a build. It shows, with one line per package, when each of this
package steps started/ended, and therefore allows to see the
sequencing of the package builds.
For a fully serialized build like we have today, this is not super
useful (except to show that everything is serialized), but it becomes
much more useful in the context of top-level parallel build.
We chose to order the graph by the time-of-configure, as it is the
closest to the actual cascade-style of a true dependency graph, which is
tiny bit more complex to achieve properly. The actual result still looks
pretty good.
The graph-build make target is extended to also generate this new
timeline graph.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- sort by start-of-configure time
- re-use existing colorsets (default or alternate)
- fix python2isms
- fix check-package
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The check_package_get_latest_version_by_distro() function analyzes the
data returned by release-monitoring.org. For two of our
packages (bento4 and qextserialport), release-monitoring.org returns
something that is a bit odd: it returns an entry with a
"stable_versions" field that contains an empty array. Our code was
ready to have or not have a "stable_versions" entry, but when it is
present, we assumed it was not an empty array. These two packages, for
some reason, break this assumption.
In order to solve this problem, this commit is more careful, and uses
the stable_versions field only if it exists and it has at least one
entry. The code is also reworked as a sequence of "if...elif...else"
to be more readable.
This fixes the following exception when running pkg-stats on the full
package set:
Task exception was never retrieved
future: <Task finished name='Task-10772' coro=<check_package_latest_version_get() done, defined at ./support/scripts/pkg-stats:532> exception=IndexError('list index out of range')>
Traceback (most recent call last):
File "./support/scripts/pkg-stats", line 535, in check_package_latest_version_get
if await check_package_get_latest_version_by_distro(session, pkg):
File "./support/scripts/pkg-stats", line 489, in check_package_get_latest_version_by_distro
version = data['stable_versions'][0] if 'stable_versions' in data else data['version'] if 'version' in data else None
IndexError: list index out of range
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: non-sequence tests as True]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
TestRust and TestRustBin has been introduced at the time when there was
no cargo package infrastructure or any package using rust compiler
(Buildroot 2018.02).
Since then the ripgrep package has been introduced, initially using
the generic package infrastructure and converted later to the cargo
package infrastructure.
Due a recent change in rust/cargo removing the cargo config file [1]
the test TestRust and TestRustBin now fail to compile since they build
an hello-world crate outside of the cargo package infrastructure
without the correct environment for cross-compiling.
Replace the 'hello-world' crate by ripgrep package and check if it
can run properly in Qemu.
Fixes tests.package.test_rust.TestRustBin:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2116202545
But doesn't fixes tests.package.test_rust.TestRust due another bug:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2116202544
[1] b6378631c2
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fixes https://gitlab.com/buildroot.org/buildroot/-/jobs/2088684091
python sample_python_pyyaml_dec.py
Traceback (most recent call last):
File "/root/sample_python_pyyaml_dec.py", line 5, in <module>
data = yaml.load(serialized)
TypeError: load() missing 1 required positional argument: 'Loader'
yaml.load() requires a loader argument since the move to version 6.0:
https://github.com/yaml/pyyaml/pull/561
The test does not need the extra functionality of load(), so instead move to
the recommended safe_load().
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 471ecea5ee (core/show-info: 'name' only applies to packages)
removed the 'name' field for rootfs (really, for non-package) entries,
thus breaking the pkg-stats processing.
We fix that by excluding any entry that has no 'name', on the assumption
that if it has no name, it is not a package.
Reported-by: Xogium on IRC
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This adds pep517(needed for flit-core to build itself) and flit python
package types.
We need to add an installer script and pass it appropriate options for
installing pep517 wheels generated by python-pypa-build during the
build stage. Unfortunately it seems pep517 does not support builds
without using the wheel format.
We also need to add a patch fixing the version parser in flit-core.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Arnout:
- fix indentation in pkg-python.mk (tabs, not spaces);
- use the new _CMD variables instead of duplicating the entire _CMDS
definitions;
- no need to filter dependencies (they're not self-referencing);
- _NEEDS_HOST_PYTHON no longer exists;
- host-python-pypa-build gets added to DEPENDENCIES automatically.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This commit add a simple test checking the reported distro name and
id are Buildroot (as reported by /etc/os-release).
Signed-off-by: Julien Olivain <ju.o@free.fr>
[Arnout: drop python2 variant]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The -z option for head was only added in coreutils 8.25, but some older
enterprise-grade distributions (e.g. the oldest still maintained RHEL 7)
only have nothing more recent than coreutils 8.22.
We fix that by using sed to remove everything that starts with the first
NULL byte, \x00.
Signed-off-by: Clayton Shotwell <clayton.shotwell@collins.com>
[yann.morin.1998@free.fr: hex is \xHH, not \xH, reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Python2 for the target is about to get removed, so drop the tests using it.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
support/testing/tests/package/test_lua_cffi.py:14:1: W391 blank line at end of file
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Allow developers to run check-package for init scripts, that call
shellcheck, without having to install the tool.
Since the docker have a fixed version of the tool, there will be no
difference between runs in different machines.
One can call:
$ utils/docker-run utils/check-package package/package/S*
$ utils/docker-run shellcheck package/package/S*
This change also allows to eventually run check-package for init scripts
in the GitLab CI.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
... so we can catch regressions on check-package.
Update to the new docker image that was pushed after the previous
commit.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
... so the unit tests for check-package can run in the GitLab CI.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
lua-sdl2 is not available on Lua 5.4, so update its test to use Lua 5.3
instead.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Some software decides based on uid/gid whether a user is a system or
normal (human) user, with different behaviour for those flavors (example
journald [2]).
So adding logic to create system-users is necessary, we take the now
common ranges from [1].
This extends the mkusers script to allow -2 for uid/gid, this argument
will take an identifier from the user range. All identifiers used up to
now should have been from the system range, so -1 is now interpreted as
a system user/group.
Note that after this commit, all the UIDs and GIDs that are created
automatically (with -1) will change. That means if there is peristent
data on an existing system that was created by such an automatic user,
it will suddenly belong to a different user. However, this could already
happen before: if a USERS line is added to a package, then other UIDs
may change as well.
Add system/user ranges as variables, and the argument for user/system
uid variable as well. Thus some magic constants could be removed, some
further occurences of -1 were replaced with equivalent logic. For
consistency, the existing MIN/MAX_UID/GID variables are renamed to
FIRST/LAST_USER_UID/GID.
Update the documentation with the new automatic ranges.
[1] - https://systemd.io/UIDS-GIDS/
[2] - https://www.freedesktop.org/software/systemd/man/journald.conf.html
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Arnout: use -1 for system users; refactor the changes a bit]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Removed a few variables, as they were only used to communicate
between the meson package and pkg-meson.mk and are not needed
anymore.
Moved cross-compilation.conf.in out of meson package.
Creating the cross-compilation.conf files for packages is now
using the original template.
To avoid duplicate code, the common sed pattern is stored in
a make variable.
Use explicit Buildroot variables for compiler tools,
and some fixes. (TARGET_LDFLAGS and TARGET_CXXFLAGS
were mixed up with PKG_TARGET_CFLAGS)
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Arnout: keep PKG_MESON_INSTALL_CROSS_CONF in
TOOLCHAIN_TARGET_FINALIZE_HOOKS]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 323ae1e681)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit add a simple test doing symmetric encryption/decryption
to check this python interface with the gpg binary is working fine.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The x86-64-v4 toolchain assumes availability of AVX512, as per the
definition of the x86-64-v4 "standard".
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Following the merge of
d6ce2a1681 ("arch/Config.in.x86: add
option for -march=x86-64") and
eeace1cc13 ("arch/Config.in.x86: add support for
x86-64-v2, x86-64-v3, x86-64-v4"), bootlin.toolchains.com now provides
toolchains targetting the x86-64, x86-64-v2, x86-64-v3 and x86-64-v4
architecture variants.
This commits modifies gen-bootlin-toolchains to support these
toolchains. It should be noted that the description for the x86-64-v3
and x86-64-v4 toolchains are for now the same, as Buildroot doesn't
yet have the options to describe the extra features that x86-64-v4
expects to find on the hardware platform.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This new test ensures that libraries and binaries generated
using Parrot Alchemy build system are correct.
Indeed, the test uses libshdata-stress.
This binary depends on libshdata.
libshdata depends on libfutils and libfutils depends on ulog.
All of these binaries and libraries are built using Alchemy.
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 1ba85b7f87 (support/download: add explicit no-hash support)
introduced the 'none' hash type, in an attempt to make hash files
mandatory, but not failing on archives localy generated, like those
for git or svn repositories, especially for those packages where a
version choice was present, which would allow for either remote
archives for which we'd have a hash or VCS trees for which we could
not have a hash for the localy generated archive.
Indeed, back in the time, we did not have a mean to generate
reproducible archives, so having a hash file without a hash for
thosel ocally generated archives would trigger an error in the
hash-checking machinery.
But now, low-and-behold, we do know how to generate those archives,
and we have a mechanism to explicitly exclude some archives from being
hash-checked (e.g. when the version string itself can be user-provided).
As such, the 'none' hash type no longer has any raison d'être, we do not
use it in-tree, and its use in a br2-external tree is most probably
inexistent (as is the use of hash files alotgether most probably).
So we simply drop the support for that.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Thomas: drop support in checkpackagelib, as reported by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
support/scripts/pkg-stats:1171:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1175:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1179:8: E713 test for membership should be 'not in'
3 E713 test for membership should be 'not in'
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/1955772278
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When debugging pkg-stats, it's quite useful to be able to disable some
features that are quite long (checking upstream URL, checking latest
version, checking CVE). This commit adds a --disable option, which can
take a comma-separated list of features to disable, such as:
./support/scripts/pkg-stats --disable url,upstream
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
