support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
import unittest
|
|
|
|
import os
|
|
|
|
import datetime
|
|
|
|
|
|
|
|
from infra.builder import Builder
|
|
|
|
from infra.emulator import Emulator
|
|
|
|
|
|
|
|
BASIC_TOOLCHAIN_CONFIG = \
|
2017-10-05 23:42:08 +02:00
|
|
|
"""
|
|
|
|
BR2_arm=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL=y
|
support/testing/infra/basetest.py: use Bootlin toolchain stable 2022.08-1
This patch updates the default toolchain used for runtime tests. The
last time this toolchain was updated was in commit
0207a65323fc73fe9570f3d7e03bbfed4ab6d477 3 years ago. Since then,
multiple things have changed:
Firstly, it used uclibc-ng as the libc whereas since commit
4057e36ca9665edd5248512e4edba2c243b8f4be, glibc is used as the default
library.
And secondly, since commit 531b2a10cda6bae78ecc141c9fe48f2024f71dbc, buildroot
dropped the support for gcc 8 and it cannot be built internally anymore.
So the testsuite was executed using a toolchain that can't be built by
the Buildroot internal toolchain backend anymore.
This new Bootlin toolchain stable 2022.08-1 is based on gcc 11.3.0,
linux headers 4.9.327, glibc 2.35 and binutils 2.38.
The previous toolchain bleeding edge 2018.11-1 is based on gcc 8.2.0,
linux headers 4.14.80, uclibc 1.0.30 and binutils 2.31.1
Nowadays Bootlin toolchains are packaged in Buildroot and we can
directly select them from BASIC_TOOLCHAIN_CONFIG and avoid
setting the toolchain parameters (BR2_TOOLCHAIN_EXTERNAL_CUSTOM...).
The switch to Glibc requires to update some tests for the following
reasons:
- TestPython3Py, TestPython3Pyc and TestPython3PyPyc has been updated
since they use the libc binary file name in their test
(uClibc: libc.so.1 vs Glibc: libc.so.6).
- TestTmux needs at least one locale to pass (as stated in tmux help
text "tmux needs a working UTF-8 locale"), so use "C.UTF-8".
- TestOpenSsh needs a toolchain >= 5.x due to a openssh issue
(Similar to: https://bugs.busybox.net/show_bug.cgi?id=13671)
Use the Bootlin toolchain bleeding-edge 2022.08-1 rhat provide
kernel headers 5.4
- TestShadow needs a toolchain >= 4.14
Use the Bootlin toolchain bleeding-edge 2022.08-1 rhat provide
kernel headers 5.4
Runtime tested on the gcc farm server.
Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-06-09 01:28:48 +02:00
|
|
|
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
|
2017-10-05 23:42:08 +02:00
|
|
|
"""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
MINIMAL_CONFIG = \
|
2017-10-05 23:42:08 +02:00
|
|
|
"""
|
|
|
|
BR2_INIT_NONE=y
|
|
|
|
BR2_SYSTEM_BIN_SH_NONE=y
|
|
|
|
# BR2_PACKAGE_BUSYBOX is not set
|
|
|
|
# BR2_TARGET_ROOTFS_TAR is not set
|
|
|
|
"""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
2017-10-05 23:42:09 +02:00
|
|
|
|
2019-02-06 04:25:29 +01:00
|
|
|
class BRConfigTest(unittest.TestCase):
|
2020-04-15 18:48:44 +02:00
|
|
|
"""Test up to the configure stage."""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
config = None
|
2019-01-29 00:22:06 +01:00
|
|
|
br2_external = list()
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
downloaddir = None
|
|
|
|
outputdir = None
|
|
|
|
logtofile = True
|
|
|
|
keepbuilds = False
|
2017-07-23 23:44:18 +02:00
|
|
|
jlevel = 0
|
2017-08-05 04:05:19 +02:00
|
|
|
timeout_multiplier = 1
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
2017-07-10 01:21:20 +02:00
|
|
|
def __init__(self, names):
|
2019-02-06 04:25:29 +01:00
|
|
|
super(BRConfigTest, self).__init__(names)
|
2017-07-10 01:21:20 +02:00
|
|
|
self.testname = self.__class__.__name__
|
2017-07-10 22:40:06 +02:00
|
|
|
self.builddir = self.outputdir and os.path.join(self.outputdir, self.testname)
|
2017-10-29 15:06:00 +01:00
|
|
|
self.config += '\nBR2_DL_DIR="{}"\n'.format(self.downloaddir)
|
2017-10-05 23:42:07 +02:00
|
|
|
self.config += "\nBR2_JLEVEL={}\n".format(self.jlevel)
|
2017-07-10 01:21:20 +02:00
|
|
|
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
def show_msg(self, msg):
|
2018-06-03 11:08:21 +02:00
|
|
|
print("{} {:40s} {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
|
|
|
|
self.testname, msg))
|
2017-10-05 23:42:09 +02:00
|
|
|
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
def setUp(self):
|
|
|
|
self.show_msg("Starting")
|
2022-12-24 10:18:12 +01:00
|
|
|
self.b = Builder(self.config, self.builddir, self.logtofile, self.jlevel)
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
if not self.keepbuilds:
|
|
|
|
self.b.delete()
|
|
|
|
|
|
|
|
if not self.b.is_finished():
|
2019-01-29 00:22:06 +01:00
|
|
|
self.b.configure(make_extra_opts=["BR2_EXTERNAL={}".format(":".join(self.br2_external))])
|
2019-02-06 04:25:29 +01:00
|
|
|
|
|
|
|
def tearDown(self):
|
|
|
|
self.show_msg("Cleaning up")
|
|
|
|
if self.b and not self.keepbuilds:
|
|
|
|
self.b.delete()
|
|
|
|
|
|
|
|
|
|
|
|
class BRTest(BRConfigTest):
|
2020-04-15 18:48:44 +02:00
|
|
|
"""Test up to the build stage and instantiate an emulator."""
|
2019-02-06 04:25:29 +01:00
|
|
|
def __init__(self, names):
|
|
|
|
super(BRTest, self).__init__(names)
|
|
|
|
self.emulator = None
|
|
|
|
|
|
|
|
def setUp(self):
|
|
|
|
super(BRTest, self).setUp()
|
|
|
|
if not self.b.is_finished():
|
|
|
|
self.show_msg("Building")
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
self.b.build()
|
|
|
|
self.show_msg("Building done")
|
|
|
|
|
2017-08-05 04:05:19 +02:00
|
|
|
self.emulator = Emulator(self.builddir, self.downloaddir,
|
|
|
|
self.logtofile, self.timeout_multiplier)
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
def tearDown(self):
|
|
|
|
if self.emulator:
|
|
|
|
self.emulator.stop()
|
2019-02-06 04:25:29 +01:00
|
|
|
super(BRTest, self).tearDown()
|
2019-11-20 14:39:51 +01:00
|
|
|
|
2019-11-25 21:30:02 +01:00
|
|
|
# Run the given 'cmd' with a 'timeout' on the target and
|
2022-12-24 10:18:11 +01:00
|
|
|
# assert that the command succeeded; on error, print the
|
|
|
|
# faulty command and its output
|
2019-11-20 14:39:51 +01:00
|
|
|
def assertRunOk(self, cmd, timeout=-1):
|
2022-12-24 10:18:11 +01:00
|
|
|
out, exit_code = self.emulator.run(cmd, timeout)
|
|
|
|
self.assertEqual(
|
|
|
|
exit_code,
|
|
|
|
0,
|
|
|
|
"\nFailed to run: {}\noutput was:\n{}".format(cmd, ' '+'\n '.join(out))
|
|
|
|
)
|