Commit Graph

58 Commits

Author SHA1 Message Date
Brandon Maier
9a780d2aca support/testing/infra/builder.py: fix missing $HOME during build
When running a test that uses host-python-setuptools using the Buildroot
Docker image, for example running the following command,

> ./utils/docker-run ./support/testing/run-tests -o output -s -k tests.package.test_python_pytest.TestPythonPy3Pytest

The build fails with the following error,

>   File "/home/blmaier/buildroot/output/TestPythonPy3Pytest/build/host-python-setuptools-69.2.0/setuptools/_distutils/dist.py", line 354, in _gen_paths
>     yield pathlib.Path('~').expanduser() / filename
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/blmaier/buildroot/output/TestPythonPy3Pytest/host/lib/python3.11/pathlib.py", line 1385, in expanduser
>     raise RuntimeError("Could not determine home directory.")
> RuntimeError: Could not determine home directory.
>
> ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

Python setuptools is looking for $HOME but failing to find it.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
(cherry picked from commit 4dafb8b5c2)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2024-06-15 20:36:51 +02:00
Romain Naour
241a8b0615 support/testing: bump prebuilt kernel to 5.10.202
As reported on the mailing list [1], TestPolkitSystemd and
TestPolkitInitd are failing since we bumped the Bootlin toolchain
2023.08 [2].

The issue is caused by expat (XML library) package detecting
arc4random_buf() introduced by glibc 2.36 [3].

With arc4random_buf() support enabled, expat hang Polkit while reading
its policy files (XML files) due to a lack of entropy on the system.

Upgrading the kernel allows to avoid such issue thanks to
random_get_entropy_fallback() introduced in 5.10.119 (backpored) [4].

Build the vexpress-v2p-ca9 (armv7) 5.10.202 using the kernel
configuration file found in /proc/config.gz but with additional kernel
options provided by SYSTEMD_LINUX_CONFIG_FIXUPS. Indeed some kernel
options requested by systemd were missing in 5.10.7 kernel.

Build the versatile-pb (armv5) kernel using the same kernel
configuration as qemu_arm_versatile_defconfig but with additional
kernel options like for vexpress-v2p-ca9. While at it, enable
IKCONFIG_PROC option to provide the kernel configuration at runtime
in /proc/config.gz.

Runtime tested:
https://gitlab.com/kubu93/buildroot/-/pipelines/1097887826 (tests.package.test_polkit.TestPolkit*)
https://gitlab.com/kubu93/buildroot/-/pipelines/1097705399 (tests.init.*)

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309948 (TestPolkitSystemd)
https://gitlab.com/buildroot.org/buildroot/-/jobs/5689309947 (TestPolkitInitd)

[1] http://lists.busybox.net/pipermail/buildroot/2023-December/680445.html
[2] 452365a385
[3] https://sourceware.org/git/?p=glibc.git;a=commit;h=6f4e0fcfa2d2b0915816a3a3a1d48b4763a7dee2
[4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fdca775081527364621857957655207d83035376

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-12-09 21:35:28 +01:00
Adam Duskett
b1d554b9a7 support/testing/infra/emulator.py: add a timeout argument for the login method
to override the current value of 60 seconds

As per a suggestion by Thomas, add a timeout argument to override the current
value of 60 seconds for the emulator.login method.

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-11-06 20:42:17 +01:00
Sebastian Weyer
f89f52168f support/testing/infra/basetest.py: use Bootlin toolchain stable 2022.08-1
This patch updates the default toolchain used for runtime tests. The
last time this toolchain was updated was in commit
0207a65323 3 years ago. Since then,
multiple things have changed:

Firstly, it used uclibc-ng as the libc whereas since commit
4057e36ca9, glibc is used as the default
library.

And secondly, since commit 531b2a10cd, buildroot
dropped the support for gcc 8 and it cannot be built internally anymore.
So the testsuite was executed using a toolchain that can't be built by
the Buildroot internal toolchain backend anymore.

This new Bootlin toolchain stable 2022.08-1 is based on gcc 11.3.0,
linux headers 4.9.327, glibc 2.35 and binutils 2.38.

The previous toolchain bleeding edge 2018.11-1 is based on gcc 8.2.0,
linux headers 4.14.80, uclibc 1.0.30 and binutils 2.31.1

Nowadays Bootlin toolchains are packaged in Buildroot and we can
directly select them from BASIC_TOOLCHAIN_CONFIG and avoid
setting the toolchain parameters (BR2_TOOLCHAIN_EXTERNAL_CUSTOM...).

The switch to Glibc requires to update some tests for the following
reasons:

- TestPython3Py, TestPython3Pyc and TestPython3PyPyc has been updated
  since they use the libc binary file name in their test
  (uClibc: libc.so.1 vs Glibc: libc.so.6).

- TestTmux needs at least one locale to pass (as stated in tmux help
  text "tmux needs a working UTF-8 locale"), so use "C.UTF-8".

- TestOpenSsh needs a toolchain >= 5.x due to a openssh issue
  (Similar to: https://bugs.busybox.net/show_bug.cgi?id=13671)
  Use the Bootlin toolchain bleeding-edge 2022.08-1 rhat provide
  kernel headers 5.4

- TestShadow needs a toolchain >= 4.14
  Use the Bootlin toolchain bleeding-edge 2022.08-1 rhat provide
  kernel headers 5.4

Runtime tested on the gcc farm server.

Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-07-12 23:22:14 +02:00
Romain Naour
e0e64e9ede support/testing/infra/emulator.py: fix typo in comment
Artefact (British) and Artifact (American) are both valid spelling
but ARTIFACTS_URL is used in the emulator code.

Surprisingly, the url actually use "artefacts"

  http://autobuild.buildroot.net/artefacts

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-06-06 16:44:16 +02:00
Yann E. MORIN
2c41fb6225 support/tests: allow top-level parallel builds
Running tests with top-level parallel builds can speed up running some
tests, expecially those that have a lot of packages like the systemd
init tests.

Trigger TLPB when the configuration enables per-package directories.

We're using the jlevel argument, which normally is used for BR2_JLEVEL
as the value for calling make -j<N> at the top-level. In fact,
BR2_JLEVEL is "unused" when using TLPB, because the top-level make
acts as the job server that distributes tokens to sub-makes (except
for the few build systems like waf or scons that don't support this),
so it's really the top-level make -j<N> that determines the level of
parallelism, and BR2_JLEVEL doesn't really have an effect.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Thomas: extend explanation a bit]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:31:41 +01:00
Yann E. MORIN
44161560dd support/tests: print failed command and output on assertRunOK error
Currently, when asserting that a command succeeded, we just capture the
return code of the command. If that is not zero, the assertion fails,
but the error message is not very splicit:
    AssertionError: 1 != 0

Replace the error message with an explicit message that dumps the failed
command, the error code, and the resulting output.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-12-27 21:40:31 +01:00
Thomas Preston
a2f11c0cbf testing/infra: Add docstrings to base test classes
It wasn't immediately obvious to me what the two Buildroot base test
classes were for, so add docstrings to explain the differences between
BRConfigTest and BRTest.

Signed-off-by: Thomas Preston <thomas.preston@codethink.co.uk>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-06 09:34:05 +01:00
Edgar Bonet
eb3ee3078a support/testing/infra/emulator.py: prevent the commands from wrapping
Traditional VT-10x terminals (and their emulators) [0] have a "magic
margins" feature that enables the last character position to be updated
without scrolling the screen: whenever a character is printed on the
last column, the cursor stays over the character, instead of moving to
the next line.

The Busybox shell, ash, attempts to defeat this feature by printing
CR,LF right after echoing a character to the last column.[1] This
doesn't play well with emulator.py. The run() method of the Emulator
class captures the output of the emulated system and assumes the first
line it reads is the echo of the command, and all subsequent lines are
the command's output. If the line made by the command + shell prompt is
longer than 80 characters, then it is echoed as two or more lines, and
all but the first one are mistaken for the command's output.

We fix this by telling the emulated system that we are using an
ultra-wide terminal with 29999 columns. Larger values would be ignored
and replaced by the default, namely 80 columns.[2]

[0] https://vt100.net/docs/vt100-ug/chapter3.html  -  DECAWM
[1] https://git.busybox.net/busybox/tree/libbb/lineedit.c?h=1_34_0#n412
[2] https://git.busybox.net/busybox/tree/libbb/xfuncs.c?h=1_34_0#n258

Signed-off-by: Edgar Bonet <bonet@grenoble.cnrs.fr>
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Co-authored-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-10-06 21:54:20 +02:00
Kory Maincent
d6d7cbb8e0 support/testing: don't fail on tests emitting invalid utf-8 sequences
When booting under EFI, grub2 will output a nice and shiny boot menu,
using extended ASCII characters (in the [0x80..0xFF] range), namely
CP437 [0], on the assumption that the VGA BIOS is a real one and has the
corresponding (and only!) font, as is the case on real hardware.

However, when run in our runtime test infrastructure, this triggers the
infamous python UnicodeDecodeError exception:

    Traceback (most recent call last):
      [...]
        emulator.login()
      File "[...]/buildroot/support/testing/infra/emulator.py", line 89, in login
        index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT],
      File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 340, in expect
        return self.expect_list(compiled_pattern_list,
      File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
        return exp.expect_loop(timeout)
      File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 111, in expect_loop
        incoming = spawn.read_nonblocking(spawn.maxread, timeout)
      File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 485, in read_nonblocking
        return super(spawn, self).read_nonblocking(size)
      File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 178, in read_nonblocking
        s = self._decoder.decode(s, final=False)
      File "/usr/lib/python3.8/codecs.py", line 322, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 0: invalid continuation byte

Grub2 is not wrong in emitting those chars, and basically we should not
expect the packages we test to always emit correct UTF-8 sequences; at
the very least, this should not cause the test infra to fail.

We fix that by telling pexpect.spawn to "fix" such invalid sequences by
replacing them with the suitable Unicode character, U+FFFD REPLACEMENT
CHARACTER.

[0] https://en.wikipedia.org/wiki/Code_page_437
[1] https://docs.python.org/3/library/codecs.html#error-handlers

Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
[yann.morin.1998@free.fr:
  - don't change encoding, use codec_errors
  - rewrite commit log accordingly
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-10-03 14:26:12 +02:00
Thomas Petazzoni
1259d40b0a support/testing/infra: add log_file_path() function
Some tests will need to grep through the build log to verify that some
features are working are expected. In order to allow them to open the
build log, we provide a new function called log_file_path(), which
returns the path to the log file if available.

We also use this function in open_log_file().

Note that open_log_file() cannot be used directly to grep through the
log file at the end of a build: because it opens in "a+" mode, it
greps starting from the end of the file.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-08-29 11:37:15 +02:00
Thomas Petazzoni
37a1af7a74 support/testing/infra: add img_round_power2() function
Since Qemu 5.1, SD card images must have a size that are a power of
two. While some filesystem (such as ext2/3/4) allow to specify the
expected size of the filesystem, others such as SquashFS do not have
this capability.

We were already extending the size of such images to the next 1 MB
boundary using "truncate -s %1M", but that is no longer sufficient. So
instead, we introduce a helper function that extends the size of an
image to the next power of two.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - use f.trunctate() rather than subprocess.call([truncate,...])
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-06-26 21:27:48 +02:00
Thomas Petazzoni
3cf2782906 support/testing/infra/emulator.py: update pre-built kernels
The pre-built vexpress kernel used by the testing infrastructure is a
4.0.0 kernel, which is getting old to be used with reasonably recent
toolchains.

This commit updates the pre-built kernels for both the versatile and
vexpress machines to 5.10.7 (they have already been put online).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-06-25 13:59:58 +02:00
Thomas Petazzoni
b0078c058a support/testing: consistently use raw strings for re.compile
Otherwise Python 3.x flake8 complains with:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:56:02 +02:00
Romain Naour
50b747f212 support/testing: check if the defconfig provided for testing is valid
Currently, the build continue even if some symbols disapear from
the generated dot config file (.config).

This patch add a new check in order to stop the test if one
of the provided symbol is missing. This must be treated as error.

For example, if a symbol disapear due to new dependency constraints.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
0207a65323 support/testing: update basic toolchain to bootlin bleeding-edge 2018.11-1
Update the toolchain being used by the testsuite infra.

The new toolchain 2018.11-1 is based on gcc 8.2, uClibc-ng 1.0.30,
linux-headers 4.14 and binutils 2.31.1.
Enable BR2_TOOLCHAIN_HAS_THREADS_DEBUG that is now required.

The old toolchain 2017.05 is based on gcc 4.9, uClibc-ng 1.0.25,
linux-headers 3.10 and binutils 2.27.

Tested with gitlab
https://gitlab.com/kubu93/buildroot/pipelines/132376578

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 22:59:34 +02:00
Pierre-Jean Texier
bda4008975 support/testing: basetest.py: fix code style
Fix these warnings:

W291 trailing whitespace

Fixes:
 - https://gitlab.com/buildroot.org/buildroot/-/jobs/360824861

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 21:40:52 +01:00
Patrick Havelange
63966e56a3 support/testing: add assertRunOk method to BRTest class
This method asserts that the given command ran successfully.
The goal is for it to be used by the different tests when needed.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-20 22:34:53 +01:00
Ricardo Martincoski
4a40d36f13 support/testing: switch to Python 3 only
Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.

In order to accomplish the change to Python 3:
 - change the shebang for run-tests;
 - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
 - when writing the downloaded binary files, explicitly open the output
   file as binary;
 - when subprocess is used to retrieve the text output from commands,
   explicitly ask for text output. For this, use 'universal_newlines'
   because 'text' was added only on Python 3.7;
 - when pexpect is used to retrieve the text output from qemu or git,
   explicitly ask for text output using 'encoding';
 - the code using csv currently follows the example in the documentation
   for the Python 2 module, change it to follow the example in the
   documentation for the Python 3 module;
 - fix the relative import for test_git.py to be Python 3 compliant.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 22:14:07 +01:00
Ricardo Martincoski
620c98a214 support/testing: factor out run_cmd_on_host
Currently many test cases call subprocess.check_output on their own.
Factor out that code to an infra method so the call get standardized.

This will be handful when switching the test infra to use Python 3.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:58 +02:00
Ricardo Martincoski
8669243da3 support/testing: allow run-tests to be called from anywhere
Currently run-tests must be called from the Buildroot top directory.

Derive the top directory from the script path, so run-tests can be called from
any path.
As a consequence the test infra will always test the repo it belongs to.

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
7acb32dabb support/testing: add builtin armv5 kernel 4.19 with entropy
More and more packages being tested by the test infra, e.g. syslog-ng,
need entropy at startup, usually reading from /dev/random.

Some test cases can also depend on a kernel version newer than the
builtin ones already provided by the test infra:
 - 3.11.0 for armv5;
 - 4.0.0 for armv7.

Add a new builtin kernel to be used by such test cases.
Add it for armv5 so most test cases that switch to use this kernel can
keep using BASIC_TOOLCHAIN_CONFIG.
Use the same kernel version and kernel config as qemu_arm_versatile plus
HW_RANDOM_VIRTIO for VirtIORNG to be usable.
Copy the actual binary file from the syslog-ng runtime test at current
master @ 29e1cb8884.

Since there is already a 'kernel-versatile' file on autobuild.buildroot.net
and we must keep it with this name for reproducibility purposes, create a
simple naming convention for newer builtin kernel images and dtb files:
kernel-<defconfig>-<kernel_series_version>
<dtb_name>-<kernel_series_version>.dtb
Pass '-device virtio-rng-pci' to qemu when this kernel is used.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Peter: use this new kernel instead of the old builtin/armv5 kernel]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Thomas Petazzoni
345c29a4b7 support/testing/infra/emulator: increase memory size used by Qemu
By default, Qemu emulates a system with 128 MB of RAM. This is not
sufficient for some test cases we have, such as TestPerlDBDmysql,
where the initramfs is quite large. Therefore, this commit extends the
RAM size emulated by Qemu to 256 MB.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/237108668

Thanks to Arnout for the analysis of the issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-22 19:28:36 +02:00
Ricardo Martincoski
0461ef4dea testing/infra: Add BRConfigTest as superclass of BRTest
The git tests don't need to do a full build, they only need to do a
configure and download and/or legal-info. More tests of that type will
be added in the future. Therefore, we want to have a test base class
that doesn't automatically do a full build in the setUp().

Add this new class as a superclass of the existing BRTest class, so we
don't need to update existing tests. Only the code in run-tests that
iterates over all subclasses of BRTest has to be adapted to use
BRConfigTest instead.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 10:40:15 +01:00
Ricardo Martincoski
4d5d0124c7 support/testing/infra/basetest: support br2-external
Some upcoming test cases can use one or more br2-external trees as
fixtures that provide packages used only in runtime tests.

Add support for br2-external into the BRTest class. Any test case can
then provide a list of paths for being used as br2-external trees
during the build of the image to test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Daniel J. Leach <dleach@belcan.com>
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Thomas: use named argument for make_extra_opts.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-29 22:16:33 +01:00
Ricardo Martincoski
07bbf1b4dd support/testing/infra/builder: configure and build with make target and environment
Make the builder able to call 'VAR1=1 make VAR2=2 target'.

Allow sending extra parameters to be added to the end of make command
line. Uses for these purposes:
 - to configure a br2-external, using the 'BR2_EXTERNAL="dir" variable.
 - to specify a make target, such as 'foo-source.'

Allow adding variables to the environment when calling make.
These added variables allow a user to override default values from BuildRoot,
such as 'BR2_DL_DIR="dl"'.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Daniel J. Leach <dleach@belcan.com>
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-29 22:14:55 +01:00
Matt Weber
b25c39ebd9 support/testing/infra: rm extra log flush in builder.py
While integrating proxy support in builder.py, a log flush
was left in the code.  This commit cleans/removes that code.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-11 12:38:45 +02:00
Matt Weber
361bc21c52 support/testing: runtest proxy support
Allow builder.py to inherit the system proxy settings from
the env if they are present.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-10 22:36:10 +02:00
Yann E. MORIN
99723554c0 support/testing: fix python syntax
Fix three issues with code style in our test infra:
  - 'print' is now a function,
  - exceptions need to be caught-assigned with the 'as' keyword,
  - old-style "%s"%() formatting is deprecated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: drop indices in format strings.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-10 15:56:25 +02:00
Ricardo Martincoski
998300c5f5 testing/infra/builder: split configure() from build()
Some test cases don't use a full build as setup, so split the build()
method into configure() and build().
It allows a test case to perform configuration at the setup stage and
the build inside the test itself.

Call this new method just before build in the BRTest base class, to keep
the current behavior for existing test cases.

This change will be needed when adding a common class to test the git
download infra.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 19:59:36 +02:00
Ricardo Martincoski
b19cb8143b testing/infra/builder: call make with empty env
Use an empty environment when calling make, but import PATH so the
buildroot tree under test can find binaries from the host machine.

Since environment variables are now ignored, move the handling of
BR2_DL_DIR to the defconfig to keep the current precedence of -d:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(DIR)
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(DIR)

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 19:59:36 +02:00
Thomas Petazzoni
193dfffa83 support/testing: use more recent toolchains
With the hard disk crash of autobuild.b.o, we lost old toolchains, so
use the latest toolchains, which have been restored.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-21 19:48:30 +02:00
Ricardo Martincoski
ef8d1f1b15 support/testing: fix code style
Fix the trivial warnings from flake8:
 - remove modules imported but unused;
 - use 2 lines before class or module level method;
 - remove blank line at end of file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:56:49 +02:00
Ricardo Martincoski
2927f412be support/testing: standardize defconfig fragments style
Change all defconfig fragments to take advantage of
"cf3cd4388a support/tests: allow properly indented config fragment".

Make each defconfig fragment:
 - start after a backslash;
 - be declared as a multi-line string literal;
 - be indented one level more than the variable that contains it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:56:06 +02:00
Ricardo Martincoski
4a1d220d9e support/testing: allow to indent ccache defconfig fragment
Move the strip out of leading spaces in defconfig fragments from the
BRTest class to the Builder class. It actually postpones the strip out,
consequentially allowing test cases to post-process the defconfig in
their own __init__ before calling the __init__ method from BRTest.

Moving this code to the Builder class also allows any new test class
that inherits from BRTest to reuse the same code even if the new class
overrides the setUp method.

At same time, prepend a newline to the jlevel handling otherwise it
would stop working for defconfig fragments that don't end in a newline.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:53:13 +02:00
Ricardo Martincoski
6e45e33f27 support/testing: allow to use a multiplier for timeouts
Add a parameter to run-tests to act as a multiplier for all timeouts of
emulator.
It can be used to avoid sporadic failures on slow host machines as well
in elastic runners on the cloud.

Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: rename multiplier to timeout_multiplier everywhere]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-08-10 10:08:06 +02:00
Ricardo Martincoski
d8ab6b2b54 testing/infra/emulator: remove qemu warnings about audio
The default audio backend for qemu is configured at compile time. It
generates annoying warning messages to qemu's stderr when running our
tests, like these:
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument

Explicitly set the audio backend to "none" at runtime to remove those
messages from our logs. There is no command line argument for this, so
use an environment variable when starting qemu.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-31 21:26:06 +02:00
Ricardo Martincoski
b6bbb0782a testing/infra/basetest: move jlevel logic to constructor
As suggested by Arnout in [1].

While at it, simplify the logic by always appending the BR2_JLEVEL and
defaulting to 0 (the value copied from Config.in is used for 5 years now
and is very unlikely to change).

[1] http://patchwork.ozlabs.org/patch/790525/

Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:51:06 +02:00
Ricardo Martincoski
78c8958619 support/testing: fix run-tests -j
Since commit cf3cd4388a the -j option is
silently ignored.

The configuration lines are processed using '\n'.join().
This function adds intervening occurrences of the separator, but the
resulting string does not end at a separator.
 >>> "n".join(["a","b"])
 'anb'
It results in a defconfig that does not end in a newline.

When BR2_JLEVEL is added by -j logic to the defconfig it ends up
concatenated to the last line of the defconfig.
 BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=yBR2_JLEVEL=7
The resulting .config has the default BR2_JLEVEL=0.

Instead of just workaround this problem by adding a newline before
BR2_JLEVEL when -j is used, make the defconfig to end in a newline since
it is a more future-proof solution.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:50:52 +02:00
Ricardo Martincoski
046f968775 testing/infra/builder: dump defconfig to log
The defconfig is composed on-the-fly by test infra + tests.

Dump it to the logfile before running 'make olddefconfig' so it can
easily analysed when debugging.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:50:45 +02:00
Andrey Smirnov
255b8ab406 testing/infra/emulator: allow to specify pexpect timeout
Some commands take more than 5 seconds to complete under QEMU, so add
provisions to allow individual unit-test to specify different duration
to avoid false negative test failures.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:37:06 +02:00
Yann E. MORIN
cf3cd4388a support/tests: allow properly indented config fragment
Currently, defining a config fragment in the runtime test infra requires
that the fragment not to be indented. This is beark, and causes grievance
when looking at the code (e.g. to fix it).

Just strip out all leading spaces/tabs when writing the configuration
lines into the config file, allowing in-line indented config fragments,
like so:

    class TestFoo(bla):
        config = bla.config + \
            """
            FOO=y
            # BAR is not set
            """

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-18 23:46:09 +02:00
Arnout Vandecappelle
704db1586c support/testing: unbreak run-tests -l
In commit b78b50465c, the initialisation
of BRTest.builddir was moved to the __init__ function. However, it is
set based on BRTest.outputdir and that is only set when the -o argument
is given to run-tests. When called as "run-tests -l", there is no -o
argument so BRTest.outputdir remains unset.

To fix, keep BRTest.builddir at None when BRTest.outputdir is None.

While we're at it, drop the direct access to the class member. If a
subclass wishes to set outputdir to something else before calling
BRTest.__init__, they are free to do so.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 23:51:33 +02:00
Arnout Vandecappelle
b78b50465c support/testing: move BRTest initialisation to __init__
BRTest's setUp() method contains a few assignments that initialize its
member variables. Since we will want to use these in test case
overrides, move them to the __init__ function.

Also allow the config member to be overridden, rather than always
taking the class member.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 17:46:25 +02:00
Arnout Vandecappelle
73e7bcdd87 support/testing: strip /usr/ part from HOST_DIR
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 17:46:22 +02:00
Ricardo Martincoski
9bb306bd64 support/testing: large timeout for login prompt
When running multiple instances of emulator in parallel, the login
prompt can take some time to appear.

Use a large timeout when waiting for the prompt to avoid random
failures.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Ricardo Martincoski
1df8042ead support/testing: allow to set BR2_JLEVEL
Let the user to override the default BR2_JLEVEL used for each testcase.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Ricardo Martincoski
abe64676d5 support/testing: fix code style in emulator
Remove unused import.
Use 2 empty lines before a class.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Ricardo Martincoski
4a622fe4b6 support/testing: use pexpect.expect directly
When using pexpect there is no need for a helper function. Just use
expect() directly everywhere.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00
Ricardo Martincoski
77b4b948fc support/testing: use pexpect.sendline directly
When using pexpect there is no need for a helper function. Just use
sendline() directly everywhere.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 16:05:14 +02:00