Commit Graph

17 Commits

Author SHA1 Message Date
Romain Naour
02fc3473e1 support/testing: bump prebuilt kernel to 5.10.202
As reported on the mailing list [1], TestPolkitSystemd and
TestPolkitInitd are failing since we bumped the Bootlin toolchain
2023.08 [2].

The issue is caused by expat (XML library) package detecting
arc4random_buf() introduced by glibc 2.36 [3].

With arc4random_buf() support enabled, expat hang Polkit while reading
its policy files (XML files) due to a lack of entropy on the system.

Upgrading the kernel allows to avoid such issue thanks to
random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4].

Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel
configuration file found in /proc/config.gz but with additional kernel
options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel
options requested by systemd were missing in 5.10.7 kernel.

Build the versatile-pb (armv5) kernel using the same kernel
configuration as qemu_arm_versatile_defconfig but with additional
kernel options like for vexpress-v2p-ca9. While at it, enable
IKCONFIG_PROC option to provide the kernel configuration at runtime
in /proc/config.gz.

Runtime tested:
https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*)
https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*)

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd)
https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd)

[1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html
[2] 452365a385
[3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2
[4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 241a8b0615)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2024-01-05 11:28:34 +01:00
Romain Naour
873306199e support/testing/run-tests: add a mode to only download emulator builtin binaries
Currently, emulator builtin binaries are downloaded after the rootfs
build by each tests running the emulator (Qemu) on armv5 or armv7.

Due to download quota on the server hosting theses binaries [1]
some GitLab jobs are failing due to "Connection reset by peer" [2]:

  ConnectionResetError: [Errno 104] Connection reset by peer

In a follow up commit, we'll add the possibility to download only once
all emulator builtin binaries to pass them as build artifacts.

As such, we will need to make sure that run-tests only download theses
binaries, and that it does not run the testsuite.

Add such a mode, with a new option, --prepare-only.

[1] http://autobuild.buildroot.net/artefacts
[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/4409032417

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-07-18 23:03:08 +02:00
Arnout Vandecappelle (Essensium/Mind)
8dce595a68 support/testing/run-tests: fix --testcases option
The --testcases option of run-tests says how many test cases to build in
parallel. It automatically derives a jlevel from it by dividing the
number of cores + 1 by the number of parallel testcases. However, this
will typically result in a fractional number. Make doesn't like
fractional numbers as argument to -j.

Convert the number to integer (rounding down).

* br2_jlevel is an int, as multiprocessing.cpu_count() is an int, so it
  will be always >=2  (cpu_count() raises an error if it can't determine
  the number of CPU, so it will always return at least 1);

* args.testcases is an int, and is checked to be >=1

So br2_jlevel + args.testcases is guaranteed to always be bigger
than or equal to args.testcases, and the division thus bigger than 1.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998@free.fr:
  - ensure division provide at least 1
  - drop the test below
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-07-27 18:20:44 +02:00
Yegor Yefremov
5abe7e4ce3 support/run-tests: reorder imports
Reorder imports using the isort utility to fix a warning from pylint3:

wrong-import-order: standard import "import multiprocessing" should be
placed before "import nose2"

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-17 10:13:08 +01:00
Yegor Yefremov
ea31dc1cd4 support/run-tests: check for empty sequences in a pythonic way
According to PEP8 empty sequences should be checked as booleans.

Fixes the following PEP8 warning:
Do not use `len(SEQUENCE)` to determine if a sequence is empty

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-16 15:38:03 +01:00
Ricardo Martincoski
4a40d36f13 support/testing: switch to Python 3 only
Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.

In order to accomplish the change to Python 3:
 - change the shebang for run-tests;
 - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
 - when writing the downloaded binary files, explicitly open the output
   file as binary;
 - when subprocess is used to retrieve the text output from commands,
   explicitly ask for text output. For this, use 'universal_newlines'
   because 'text' was added only on Python 3.7;
 - when pexpect is used to retrieve the text output from qemu or git,
   explicitly ask for text output using 'encoding';
 - the code using csv currently follows the example in the documentation
   for the Python 2 module, change it to follow the example in the
   documentation for the Python 3 module;
 - fix the relative import for test_git.py to be Python 3 compliant.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 22:14:07 +01:00
Ricardo Martincoski
0461ef4dea testing/infra: Add BRConfigTest as superclass of BRTest
The git tests don't need to do a full build, they only need to do a
configure and download and/or legal-info. More tests of that type will
be added in the future. Therefore, we want to have a test base class
that doesn't automatically do a full build in the setUp().

Add this new class as a superclass of the existing BRTest class, so we
don't need to update existing tests. Only the code in run-tests that
iterates over all subclasses of BRTest has to be adapted to use
BRConfigTest instead.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 10:40:15 +01:00
Yann E. MORIN
99723554c0 support/testing: fix python syntax
Fix three issues with code style in our test infra:
  - 'print' is now a function,
  - exceptions need to be caught-assigned with the 'as' keyword,
  - old-style "%s"%() formatting is deprecated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: drop indices in format strings.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-10 15:56:25 +02:00
Ricardo Martincoski
b19cb8143b testing/infra/builder: call make with empty env
Use an empty environment when calling make, but import PATH so the
buildroot tree under test can find binaries from the host machine.

Since environment variables are now ignored, move the handling of
BR2_DL_DIR to the defconfig to keep the current precedence of -d:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(DIR)
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(DIR)

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 19:59:36 +02:00
Yann E. MORIN
33f3cb8a87 support/run-tests: export download dir
Currently, the download directory, when specified with the -d option, is
only used to store the files downloaded by the testing infra, not those
downloaded by Buildroot.

So, we end up with this situation:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(TOP_DIR)/dl
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(BR2_DL_DIR)

This is not very consistent.

We change the behaviour so that the value of -d always takes precedence,
and is used by Buildroot as well, giving this new behaviour:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(DIR)
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(DIR)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-10 23:33:07 +02:00
Ricardo Martincoski
ef8d1f1b15 support/testing: fix code style
Fix the trivial warnings from flake8:
 - remove modules imported but unused;
 - use 2 lines before class or module level method;
 - remove blank line at end of file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:56:49 +02:00
Yann E. MORIN
ebe61135c9 support/tests: use relative test_dir
When we list the available tests, we use test_dir, which is set from
the path of the script. However, when we run the tests, we use the
hard-coded path.

Ditto to find the config file.

For consistency, always use test_dir.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-09-05 21:09:06 +02:00
Ricardo Martincoski
6e45e33f27 support/testing: allow to use a multiplier for timeouts
Add a parameter to run-tests to act as a multiplier for all timeouts of
emulator.
It can be used to avoid sporadic failures on slow host machines as well
in elastic runners on the cloud.

Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: rename multiplier to timeout_multiplier everywhere]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-08-10 10:08:06 +02:00
Ricardo Martincoski
f98b93daaa support/testing: run testcases in parallel
Let the user to pass -t to set the number of testcases to run
simultaneously.

When -j is not specified, calculate it to split the available cores
between the simultaneous testcases.

Example of auto calculated -j for cpu_count 8:
  -t -j  total
   1  9  9
   2  4  8
   3  3  9
   4  2  8
 >=5  1  t

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Ricardo Martincoski
1df8042ead support/testing: allow to set BR2_JLEVEL
Let the user to override the default BR2_JLEVEL used for each testcase.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Luca Ceresoli
fa3c5cad44 support/testing/run-tests: help: put the one-letter form before the long form
This is what the manpages usually do, and what Python does with the
automatically-added -h/--help parameter:

Before the change:
  $ ./support/testing/run-tests
    [...]
  optional arguments:
    -h, --help            show this help message and exit
    --list, -l            list of available test cases
    --all, -a             execute all test cases

After the change:
  $ ./support/testing/run-tests
    [...]
  optional arguments:
    -h, --help            show this help message and exit
    -l, --list            list of available test cases
    -a, --all             execute all test cases

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-05-17 21:56:49 +02:00
Thomas Petazzoni
a732fb222b support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.

This core infrastructure commit includes the following aspects:

 - A base test class, called BRTest, defined in
   support/testing/infra/basetest.py. This base test class inherited
   from the Python provided unittest.TestCase, and must be subclassed by
   all Buildroot test cases.

   Its main purpose is to provide the Python unittest setUp() and
   tearDown() methods. In our case, setUp() takes care of building the
   Buildroot system described in the test case, and instantiate the
   Emulator object in case runtime testing is needed. The tearDown()
   method simply cleans things up (stop the emulator, remove the output
   directory).

 - A Builder class, defined in support/testing/infra/builder.py, simply
   responsible for building the Buildroot system in each test case.

 - An Emulator class, defined in support/testing/infra/emulator.py,
   responsible for running the generated system under Qemu, allowing
   each test case to run arbitrary commands inside the emulated system.

 - A run-tests script, which is the entry point to start the tests.

Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-05-07 22:04:54 +02:00