support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
import unittest
|
|
|
|
import os
|
|
|
|
import datetime
|
|
|
|
|
|
|
|
from infra.builder import Builder
|
|
|
|
from infra.emulator import Emulator
|
|
|
|
|
|
|
|
BASIC_TOOLCHAIN_CONFIG = \
|
2017-10-05 23:42:08 +02:00
|
|
|
"""
|
|
|
|
BR2_arm=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
|
2020-04-02 23:57:18 +02:00
|
|
|
BR2_TOOLCHAIN_EXTERNAL_URL="https://toolchains.bootlin.com/downloads/releases/toolchains/armv5-eabi/tarballs/armv5-eabi--uclibc--bleeding-edge-2018.11-1.tar.bz2"
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
|
|
|
|
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_14=y
|
2017-10-05 23:42:08 +02:00
|
|
|
BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
|
2020-04-02 23:57:18 +02:00
|
|
|
BR2_TOOLCHAIN_HAS_THREADS_DEBUG=y
|
2017-10-05 23:42:08 +02:00
|
|
|
BR2_TOOLCHAIN_EXTERNAL_CXX=y
|
|
|
|
"""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
MINIMAL_CONFIG = \
|
2017-10-05 23:42:08 +02:00
|
|
|
"""
|
|
|
|
BR2_INIT_NONE=y
|
|
|
|
BR2_SYSTEM_BIN_SH_NONE=y
|
|
|
|
# BR2_PACKAGE_BUSYBOX is not set
|
|
|
|
# BR2_TARGET_ROOTFS_TAR is not set
|
|
|
|
"""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
2017-10-05 23:42:09 +02:00
|
|
|
|
2019-02-06 04:25:29 +01:00
|
|
|
class BRConfigTest(unittest.TestCase):
|
2020-04-15 18:48:44 +02:00
|
|
|
"""Test up to the configure stage."""
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
config = None
|
2019-01-29 00:22:06 +01:00
|
|
|
br2_external = list()
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
downloaddir = None
|
|
|
|
outputdir = None
|
|
|
|
logtofile = True
|
|
|
|
keepbuilds = False
|
2017-07-23 23:44:18 +02:00
|
|
|
jlevel = 0
|
2017-08-05 04:05:19 +02:00
|
|
|
timeout_multiplier = 1
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
2017-07-10 01:21:20 +02:00
|
|
|
def __init__(self, names):
|
2019-02-06 04:25:29 +01:00
|
|
|
super(BRConfigTest, self).__init__(names)
|
2017-07-10 01:21:20 +02:00
|
|
|
self.testname = self.__class__.__name__
|
2017-07-10 22:40:06 +02:00
|
|
|
self.builddir = self.outputdir and os.path.join(self.outputdir, self.testname)
|
2017-10-29 15:06:00 +01:00
|
|
|
self.config += '\nBR2_DL_DIR="{}"\n'.format(self.downloaddir)
|
2017-10-05 23:42:07 +02:00
|
|
|
self.config += "\nBR2_JLEVEL={}\n".format(self.jlevel)
|
2017-07-10 01:21:20 +02:00
|
|
|
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
def show_msg(self, msg):
|
2018-06-03 11:08:21 +02:00
|
|
|
print("{} {:40s} {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
|
|
|
|
self.testname, msg))
|
2017-10-05 23:42:09 +02:00
|
|
|
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
def setUp(self):
|
|
|
|
self.show_msg("Starting")
|
2017-07-23 23:44:18 +02:00
|
|
|
self.b = Builder(self.config, self.builddir, self.logtofile)
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
if not self.keepbuilds:
|
|
|
|
self.b.delete()
|
|
|
|
|
|
|
|
if not self.b.is_finished():
|
2019-01-29 00:22:06 +01:00
|
|
|
self.b.configure(make_extra_opts=["BR2_EXTERNAL={}".format(":".join(self.br2_external))])
|
2019-02-06 04:25:29 +01:00
|
|
|
|
|
|
|
def tearDown(self):
|
|
|
|
self.show_msg("Cleaning up")
|
|
|
|
if self.b and not self.keepbuilds:
|
|
|
|
self.b.delete()
|
|
|
|
|
|
|
|
|
|
|
|
class BRTest(BRConfigTest):
|
2020-04-15 18:48:44 +02:00
|
|
|
"""Test up to the build stage and instantiate an emulator."""
|
2019-02-06 04:25:29 +01:00
|
|
|
def __init__(self, names):
|
|
|
|
super(BRTest, self).__init__(names)
|
|
|
|
self.emulator = None
|
|
|
|
|
|
|
|
def setUp(self):
|
|
|
|
super(BRTest, self).setUp()
|
|
|
|
if not self.b.is_finished():
|
|
|
|
self.show_msg("Building")
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
self.b.build()
|
|
|
|
self.show_msg("Building done")
|
|
|
|
|
2017-08-05 04:05:19 +02:00
|
|
|
self.emulator = Emulator(self.builddir, self.downloaddir,
|
|
|
|
self.logtofile, self.timeout_multiplier)
|
support/testing: core testing infrastructure
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
|
|
|
|
|
|
|
def tearDown(self):
|
|
|
|
if self.emulator:
|
|
|
|
self.emulator.stop()
|
2019-02-06 04:25:29 +01:00
|
|
|
super(BRTest, self).tearDown()
|
2019-11-20 14:39:51 +01:00
|
|
|
|
2019-11-25 21:30:02 +01:00
|
|
|
# Run the given 'cmd' with a 'timeout' on the target and
|
2019-11-20 14:39:51 +01:00
|
|
|
# assert that the command succeeded
|
|
|
|
def assertRunOk(self, cmd, timeout=-1):
|
|
|
|
_, exit_code = self.emulator.run(cmd, timeout)
|
|
|
|
self.assertEqual(exit_code, 0)
|