kumquat-buildroot/support/testing/infra/emulator.py

129 lines
5.0 KiB
Python
Raw Normal View History

import pexpect
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
import infra
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
class Emulator(object):
def __init__(self, builddir, downloaddir, logtofile, timeout_multiplier):
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
self.qemu = None
self.downloaddir = downloaddir
self.logfile = infra.open_log_file(builddir, "run", logtofile)
# We use elastic runners on the cloud to runs our tests. Those runners
# can take a long time to run the emulator. Use a timeout multiplier
# when running the tests to avoid sporadic failures.
self.timeout_multiplier = timeout_multiplier
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
# Start Qemu to boot the system
#
# arch: Qemu architecture to use
#
# kernel: path to the kernel image, or the special string
# 'builtin'. 'builtin' means a pre-built kernel image will be
# downloaded from ARTIFACTS_URL and suitable options are
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
# automatically passed to qemu and added to the kernel cmdline. So
# far only armv5, armv7 and i386 builtin kernels are available.
# If None, then no kernel is used, and we assume a bootable device
# will be specified.
#
# kernel_cmdline: array of kernel arguments to pass to Qemu -append option
#
# options: array of command line options to pass to Qemu
#
def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
if arch in ["armv7", "armv5"]:
qemu_arch = "arm"
else:
qemu_arch = arch
qemu_cmd = ["qemu-system-{}".format(qemu_arch),
"-serial", "stdio",
"-display", "none",
"-m", "256"]
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
if options:
qemu_cmd += options
if kernel_cmdline is None:
kernel_cmdline = []
if kernel:
if kernel == "builtin":
if arch in ["armv7", "armv5"]:
kernel_cmdline.append("console=ttyAMA0")
if arch == "armv7":
kernel = infra.download(self.downloaddir,
support/testing: bump prebuilt kernel to 5.10.202 As reported on the mailing list [1], TestPolkitSystemd and TestPolkitInitd are failing since we bumped the Bootlin toolchain 2023.08 [2]. The issue is caused by expat (XML library) package detecting arc4random_buf() introduced by glibc 2.36 [3]. With arc4random_buf() support enabled, expat hang Polkit while reading its policy files (XML files) due to a lack of entropy on the system. Upgrading the kernel allows to avoid such issue thanks to random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4]. Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel configuration file found in /proc/config.gz but with additional kernel options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel options requested by systemd were missing in 5.10.7 kernel. Build the versatile-pb (armv5) kernel using the same kernel configuration as qemu_arm_versatile_defconfig but with additional kernel options like for vexpress-v2p-ca9. While at it, enable IKCONFIG_PROC option to provide the kernel configuration at runtime in /proc/config.gz. Runtime tested: https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*) https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*) Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd) https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd) [1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html [2] 452365a385ef35e808eb2eb669d1e3227c7399dd [3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2 [4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376 Signed-off-by: Romain Naour <romain.naour@smile.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-12-06 23:44:20 +01:00
"kernel-vexpress-5.10.202")
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
dtb = infra.download(self.downloaddir,
support/testing: bump prebuilt kernel to 5.10.202 As reported on the mailing list [1], TestPolkitSystemd and TestPolkitInitd are failing since we bumped the Bootlin toolchain 2023.08 [2]. The issue is caused by expat (XML library) package detecting arc4random_buf() introduced by glibc 2.36 [3]. With arc4random_buf() support enabled, expat hang Polkit while reading its policy files (XML files) due to a lack of entropy on the system. Upgrading the kernel allows to avoid such issue thanks to random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4]. Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel configuration file found in /proc/config.gz but with additional kernel options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel options requested by systemd were missing in 5.10.7 kernel. Build the versatile-pb (armv5) kernel using the same kernel configuration as qemu_arm_versatile_defconfig but with additional kernel options like for vexpress-v2p-ca9. While at it, enable IKCONFIG_PROC option to provide the kernel configuration at runtime in /proc/config.gz. Runtime tested: https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*) https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*) Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd) https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd) [1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html [2] 452365a385ef35e808eb2eb669d1e3227c7399dd [3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2 [4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376 Signed-off-by: Romain Naour <romain.naour@smile.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-12-06 23:44:20 +01:00
"vexpress-v2p-ca9-5.10.202.dtb")
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
qemu_cmd += ["-dtb", dtb]
qemu_cmd += ["-M", "vexpress-a9"]
elif arch == "armv5":
kernel = infra.download(self.downloaddir,
support/testing: bump prebuilt kernel to 5.10.202 As reported on the mailing list [1], TestPolkitSystemd and TestPolkitInitd are failing since we bumped the Bootlin toolchain 2023.08 [2]. The issue is caused by expat (XML library) package detecting arc4random_buf() introduced by glibc 2.36 [3]. With arc4random_buf() support enabled, expat hang Polkit while reading its policy files (XML files) due to a lack of entropy on the system. Upgrading the kernel allows to avoid such issue thanks to random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4]. Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel configuration file found in /proc/config.gz but with additional kernel options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel options requested by systemd were missing in 5.10.7 kernel. Build the versatile-pb (armv5) kernel using the same kernel configuration as qemu_arm_versatile_defconfig but with additional kernel options like for vexpress-v2p-ca9. While at it, enable IKCONFIG_PROC option to provide the kernel configuration at runtime in /proc/config.gz. Runtime tested: https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*) https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*) Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd) https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd) [1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html [2] 452365a385ef35e808eb2eb669d1e3227c7399dd [3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2 [4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376 Signed-off-by: Romain Naour <romain.naour@smile.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-12-06 23:44:20 +01:00
"kernel-versatile-5.10.202")
dtb = infra.download(self.downloaddir,
support/testing: bump prebuilt kernel to 5.10.202 As reported on the mailing list [1], TestPolkitSystemd and TestPolkitInitd are failing since we bumped the Bootlin toolchain 2023.08 [2]. The issue is caused by expat (XML library) package detecting arc4random_buf() introduced by glibc 2.36 [3]. With arc4random_buf() support enabled, expat hang Polkit while reading its policy files (XML files) due to a lack of entropy on the system. Upgrading the kernel allows to avoid such issue thanks to random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4]. Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel configuration file found in /proc/config.gz but with additional kernel options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel options requested by systemd were missing in 5.10.7 kernel. Build the versatile-pb (armv5) kernel using the same kernel configuration as qemu_arm_versatile_defconfig but with additional kernel options like for vexpress-v2p-ca9. While at it, enable IKCONFIG_PROC option to provide the kernel configuration at runtime in /proc/config.gz. Runtime tested: https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*) https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*) Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd) https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd) [1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html [2] 452365a385ef35e808eb2eb669d1e3227c7399dd [3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2 [4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376 Signed-off-by: Romain Naour <romain.naour@smile.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-12-06 23:44:20 +01:00
"versatile-pb-5.10.202.dtb")
qemu_cmd += ["-dtb", dtb]
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
qemu_cmd += ["-M", "versatilepb"]
qemu_cmd += ["-device", "virtio-rng-pci"]
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
qemu_cmd += ["-kernel", kernel]
if kernel_cmdline:
qemu_cmd += ["-append", " ".join(kernel_cmdline)]
self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:],
timeout=5 * self.timeout_multiplier,
support/testing: switch to Python 3 only Python 2.7 will not be maintained past 2020. Many scripts on the tree are used during the build and should keep Python 2 compatibility for a while. This is not the case for the runtime test infra. It's meant to be run in modern distros only, so it can safely switch to support Python 3 only. An advantage of this approach is to have less scenarios to test in. Otherwise every change to the test infra or runtime tests would need to be tested against both versions of the interpreter, increasing the effort of the developers, to ensure the compatibility to Python 2 was not broken. In order to accomplish the change to Python 3: - change the shebang for run-tests; - use Python 3 urllib as a drop-in replacement for Python 2 urllib2; - when writing the downloaded binary files, explicitly open the output file as binary; - when subprocess is used to retrieve the text output from commands, explicitly ask for text output. For this, use 'universal_newlines' because 'text' was added only on Python 3.7; - when pexpect is used to retrieve the text output from qemu or git, explicitly ask for text output using 'encoding'; - the code using csv currently follows the example in the documentation for the Python 2 module, change it to follow the example in the documentation for the Python 3 module; - fix the relative import for test_git.py to be Python 3 compliant. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Tested-by: Romain Naour <romain.naour@smile.fr> Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com> Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-27 14:37:00 +01:00
encoding='utf-8',
support/testing: don't fail on tests emitting invalid utf-8 sequences When booting under EFI, grub2 will output a nice and shiny boot menu, using extended ASCII characters (in the [0x80..0xFF] range), namely CP437 [0], on the assumption that the VGA BIOS is a real one and has the corresponding (and only!) font, as is the case on real hardware. However, when run in our runtime test infrastructure, this triggers the infamous python UnicodeDecodeError exception: Traceback (most recent call last): [...] emulator.login() File "[...]/buildroot/support/testing/infra/emulator.py", line 89, in login index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT], File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 340, in expect return self.expect_list(compiled_pattern_list, File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 369, in expect_list return exp.expect_loop(timeout) File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 111, in expect_loop incoming = spawn.read_nonblocking(spawn.maxread, timeout) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 485, in read_nonblocking return super(spawn, self).read_nonblocking(size) File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 178, in read_nonblocking s = self._decoder.decode(s, final=False) File "/usr/lib/python3.8/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 0: invalid continuation byte Grub2 is not wrong in emitting those chars, and basically we should not expect the packages we test to always emit correct UTF-8 sequences; at the very least, this should not cause the test infra to fail. We fix that by telling pexpect.spawn to "fix" such invalid sequences by replacing them with the suitable Unicode character, U+FFFD REPLACEMENT CHARACTER. [0] https://en.wikipedia.org/wiki/Code_page_437 [1] https://docs.python.org/3/library/codecs.html#error-handlers Signed-off-by: Kory Maincent <kory.maincent@bootlin.com> [yann.morin.1998@free.fr: - don't change encoding, use codec_errors - rewrite commit log accordingly ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-09-23 17:57:24 +02:00
codec_errors='replace',
env={"QEMU_AUDIO_DRV": "none"})
# We want only stdout into the log to avoid double echo
self.qemu.logfile_read = self.logfile
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
# Wait for the login prompt to appear, and then login as root with
# the provided password, or no password if not specified.
def login(self, password=None, timeout=60):
# The login prompt can take some time to appear when running multiple
# instances in parallel, so set the timeout to a large value
index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT],
timeout=timeout * self.timeout_multiplier)
if index != 0:
self.logfile.write("==> System does not boot")
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
raise SystemError("System does not boot")
self.qemu.sendline("root")
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
if password:
self.qemu.expect("Password:")
self.qemu.sendline(password)
index = self.qemu.expect(["# ", pexpect.TIMEOUT])
if index != 0:
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
raise SystemError("Cannot login")
self.run("dmesg -n 1")
support/testing/infra/emulator.py: prevent the commands from wrapping Traditional VT-10x terminals (and their emulators) [0] have a "magic margins" feature that enables the last character position to be updated without scrolling the screen: whenever a character is printed on the last column, the cursor stays over the character, instead of moving to the next line. The Busybox shell, ash, attempts to defeat this feature by printing CR,LF right after echoing a character to the last column.[1] This doesn't play well with emulator.py. The run() method of the Emulator class captures the output of the emulated system and assumes the first line it reads is the echo of the command, and all subsequent lines are the command's output. If the line made by the command + shell prompt is longer than 80 characters, then it is echoed as two or more lines, and all but the first one are mistaken for the command's output. We fix this by telling the emulated system that we are using an ultra-wide terminal with 29999 columns. Larger values would be ignored and replaced by the default, namely 80 columns.[2] [0] https://vt100.net/docs/vt100-ug/chapter3.html - DECAWM [1] https://git.busybox.net/busybox/tree/libbb/lineedit.c?h=1_34_0#n412 [2] https://git.busybox.net/busybox/tree/libbb/xfuncs.c?h=1_34_0#n258 Signed-off-by: Edgar Bonet <bonet@grenoble.cnrs.fr> Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Co-authored-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-10-05 19:17:58 +02:00
# Prevent the shell from wrapping the commands at 80 columns.
self.run("stty columns 29999")
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
# Run the given 'cmd' with a 'timeout' on the target
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
# return a tuple (output, exit_code)
def run(self, cmd, timeout=-1):
self.qemu.sendline(cmd)
if timeout != -1:
timeout *= self.timeout_multiplier
self.qemu.expect("# ", timeout=timeout)
# Remove double carriage return from qemu stdout so str.splitlines()
# works as expected.
output = self.qemu.before.replace("\r\r", "\r").splitlines()[1:]
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
self.qemu.sendline("echo $?")
self.qemu.expect("# ")
exit_code = self.qemu.before.splitlines()[2]
support/testing: core testing infrastructure This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-03-20 21:36:50 +01:00
exit_code = int(exit_code)
return output, exit_code
def stop(self):
if self.qemu is None:
return
self.qemu.terminate(force=True)