When we list the available tests, we use test_dir, which is set from
the path of the script. However, when we run the tests, we use the
hard-coded path.
Ditto to find the config file.
For consistency, always use test_dir.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add a parameter to run-tests to act as a multiplier for all timeouts of
emulator.
It can be used to avoid sporadic failures on slow host machines as well
in elastic runners on the cloud.
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: rename multiplier to timeout_multiplier everywhere]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Let the user to pass -t to set the number of testcases to run
simultaneously.
When -j is not specified, calculate it to split the available cores
between the simultaneous testcases.
Example of auto calculated -j for cpu_count 8:
-t -j total
1 9 9
2 4 8
3 3 9
4 2 8
>=5 1 t
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Let the user to override the default BR2_JLEVEL used for each testcase.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This is what the manpages usually do, and what Python does with the
automatically-added -h/--help parameter:
Before the change:
$ ./support/testing/run-tests
[...]
optional arguments:
-h, --help show this help message and exit
--list, -l list of available test cases
--all, -a execute all test cases
After the change:
$ ./support/testing/run-tests
[...]
optional arguments:
-h, --help show this help message and exit
-l, --list list of available test cases
-a, --all execute all test cases
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
