Commit Graph

73 Commits

Author SHA1 Message Date
Yann E. MORIN
99723554c0 support/testing: fix python syntax
Fix three issues with code style in our test infra:
  - 'print' is now a function,
  - exceptions need to be caught-assigned with the 'as' keyword,
  - old-style "%s"%() formatting is deprecated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: drop indices in format strings.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-10 15:56:25 +02:00
Eric Le Bihan
09a5eb427a support/testing: set $USER in rust tests
When the run-time tests to build rust and rust-bin packages are run via Docker,
the $USER environment variable is not set, which makes cargo fail when
initializing the test project.

So add it to make cargo happy.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-29 09:57:32 +02:00
Thomas Petazzoni
ac260a2ace support/testing: fix Marvell ATF source code
The version of the ARM Trusted Firmware from Marvell was a Git branch,
not a Git commit, leading to unreproducible results. So let's use a
Git commit instead, which is the latest available from the branch that
was previously used.

More specifically, this branch has recently seen a fix that is needed
for ATF to build properly with recent gcc versions:

  c96ec59f8b

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-25 21:28:47 +02:00
Thomas Petazzoni
e74a7cd1e0 support/testing: fix ATF Vexpress test case
This test case currently fails to build with:

./build/juno/release/bl1/context_mgmt.o: In function `cm_prepare_el3_exit':
context_mgmt.c:(.text.cm_prepare_el3_exit+0x54): undefined reference to `cm_set_next_context'
context_mgmt.c:(.text.cm_prepare_el3_exit+0x54): relocation truncated to fit: R_AARCH64_JUMP26 against undefined symbol `cm_set_next_context'

This issue has been fixed upstream in commit
10c252c14b7f446c0b49ef1aafbd5d37804577dd, available since v1.3. So
while we bump, let's bump to the latest version of ATF, v1.5.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/64360659

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-25 21:28:18 +02:00
Ricardo Martincoski
41a7c72c21 support/testing: add test for syslog-ng
Check:
 - the daemon is started;
 - a message is collected;
 - the daemon does not issue a warning message on startup.

When the .conf file version does not match the package version a warning
message shows up on serial on every boot. This message is generated by
syslog-ng before it is running, so it is not logged to
/var/log/messages. So in order to test the message is generated, restart
the server. It makes the message appears on /var/log/messages (since the
server is already running) where its existence can be easily tested
using grep.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-19 23:16:15 +02:00
Ricardo Martincoski
998300c5f5 testing/infra/builder: split configure() from build()
Some test cases don't use a full build as setup, so split the build()
method into configure() and build().
It allows a test case to perform configuration at the setup stage and
the build inside the test itself.

Call this new method just before build in the BRTest base class, to keep
the current behavior for existing test cases.

This change will be needed when adding a common class to test the git
download infra.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 19:59:36 +02:00
Ricardo Martincoski
b19cb8143b testing/infra/builder: call make with empty env
Use an empty environment when calling make, but import PATH so the
buildroot tree under test can find binaries from the host machine.

Since environment variables are now ignored, move the handling of
BR2_DL_DIR to the defconfig to keep the current precedence of -d:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(DIR)
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(DIR)

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 19:59:36 +02:00
Yann E. MORIN
bb2a57a17a fs: run packages' filesystem hooks in a copy of target/
Currently, some packages may register hooks to be run just before and
just after the generic tarball image is generated, because they need to
prepare the filesystem for read-only or read-write operation.

However, this means that, if any of the hooks or the image generation
fails, the target directory is left in a dangling, inconsistent state.

We fix that by doing a copy of target/, run the hooks on that copy,
generate the generic tarball image out of that, and get rid of the copy.

This way, we can guarantee consistency of the target directory, and we
can even ditch support for post-fs hooks (those that restore target/).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-31 20:53:06 +02:00
Ricardo Martincoski
d44816f611 test_python_cryptography: fix code style
Fix these warnings:
E122 continuation line missing indentation or outdented
E301 expected 1 blank line, found 0
E302 expected 2 blank lines, found 1
F401 'os' imported but unused

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-13 22:28:05 +01:00
Peter Korsgaard
92b8bd0879 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-05 21:04:14 +01:00
Yann E. MORIN
69781ebb50 support/tests: enhance the runtime systemd tests
Recent systemd bump has broken DBus dameon and DBus applications can no
longer find the daemon. So we want to catch those kind of failures
early.

We also want to check that the system as a whole is stable: no unit
should be failed.

Finally, ensure that we can read the jounrnal, even when we are doing our
tricks on read-only systems.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-04 21:10:23 +01:00
Ricardo Martincoski
2b0ca5749c support/testing: fix job check-gitlab-ci.yml
Currently 'run-tests -l' is broken. It breaks 'make .gitlab-ci.yml' that
in turn breaks the job in GitLab.

TestRustBase is not a test case by itself, so it can't have a method
with the name starting with "test_" otherwise nose2 assumes it is a test
case.
Move the test_run method from the base class to the derived classes.

While at it, update .gitlab-ci.yml with the new test cases.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/52000035

Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-02-13 23:12:00 +01:00
Yegor Yefremov
ecfde09985 support/testing: add python-cryptography tests
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-02-06 15:10:39 +01:00
Eric Le Bihan
e9429539ec support/testing: add tests for Rust
To test the support for the Rust language, the following tests are added:

- building Rust compiler and Cargo from source.
- installing a pre-built Rust compiler and building Cargo from source.

For each test, a Rust test program is built and installed in the root file
system of a ARM vexpress QEMU system. The test is declared OK if the program can
be run properly from the test system.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-02-05 14:57:48 +01:00
Ricardo Martincoski
a3bb315e42 testing/tests/boot/test_atf: fix code style
Fix these warnings:
E122 continuation line missing indentation or outdented
E302 expected 2 blank lines, found 1
F401 'os' imported but unused

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Yann E. MORIN
02fc52fe59 tests: add test for post-fakeroot script
Currently, only post-build and post-image scripts were tested, each with
their own test-script.

The two test-scripts only differ in the name of the log file they
create, and it is based on the name of the script, so it is easy to
share the script.

This allows us to easily re-use it for testing post-fakeroot scripts.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-01-15 21:37:45 +01:00
Yann E. MORIN
158e832aac fs/iso9660: fix transparent (de)compression
It needs mkzftree from zisofs-tools, so we add a dependency to it, and
we call that one explicitly (to avoid using the one from the host in
PATH).

It also needs the the uncompressed kernel image, but because it is
already in target/ so it gets compressed by mkzftree. We have two
options:
  - compress everything but the kernel image,
  - compress everything, kernel included, and recopy it later.

We choose the latter, because it is the simplest solution. So, we always
define the kernel-copy hook, but only register it when needed.

Finally, it needs a kernel with support for transparent
(de)compression, so we update the existing test config.

Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-02 21:57:43 +01:00
Yann E. MORIN
290b4cfe17 core/tests: add test for compressed iso9660
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: really set the transparent compression option, take into
account the renaming of the option.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-12-31 18:20:14 +01:00
Thomas Petazzoni
7794ce5089 support/testing: TestATFMarvell needs BR2_TARGET_UBOOT_NEEDS_DTC=y
The Marvell U-Boot needs DTC to build, so the test case fails to build
on systems where dtc isn't available. We add
BR2_TARGET_UBOOT_NEEDS_DTC=y to make sure that we build our own DTC.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/44126707

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-12-12 09:36:30 +01:00
Thomas Petazzoni
8cf3ce04e9 support/testing: add tests for ATF
These new tests only do build tests, but allow to quickly verify that
the ATF/U-Boot combinations for vexpress, Allwinner and Marvell
platforms all continue to build properly.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-12-08 19:00:59 +01:00
Thomas Petazzoni
eb5b132376 support/testing: update ISO9660 test case Linux kernel
The Linux 4.0 kernel doesn't build with gcc 6.x, which is used since
the toolchain update in commit
193dfffa83 ("support/testing: use more
recent toolchains"). So let's update to Linux 4.11 instead (like the
existing Qemu x86 defconfig does), and update the kernel configuration
file accordingly.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-21 21:16:25 +02:00
Thomas Petazzoni
193dfffa83 support/testing: use more recent toolchains
With the hard disk crash of autobuild.b.o, we lost old toolchains, so
use the latest toolchains, which have been restored.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-21 19:48:30 +02:00
Yann E. MORIN
33f3cb8a87 support/run-tests: export download dir
Currently, the download directory, when specified with the -d option, is
only used to store the files downloaded by the testing infra, not those
downloaded by Buildroot.

So, we end up with this situation:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(TOP_DIR)/dl
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(BR2_DL_DIR)

This is not very consistent.

We change the behaviour so that the value of -d always takes precedence,
and is used by Buildroot as well, giving this new behaviour:

    BR2_DL_DIR  | -d DIR   | test downloads   | BR downloads
    ------------+----------+------------------+--------------
    unset       | unset    | [error]          | [error]
    unset       | set      | in $(DIR)        | in $(DIR)
    set         | unset    | in $(BR2_DL_DIR) | in $(BR2_DL_DIR)
    set         | set      | in $(DIR)        | in $(DIR)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-10 23:33:07 +02:00
Ricardo Martincoski
022d8df1b5 support/testing: fix remaining code style
Fix the remaining code style warnings from flake8:
 - properly indent continuation lines;
 - use proper code to test a parameter is not None.

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 19:06:05 +02:00
Ricardo Martincoski
847d6d1e85 testing/tests/init: use lowercase method names
Use method naming convention from PEP8 as other test cases already do.

sed \
  -e 's,startEmulator,start_emulator,g' \
  -e 's,checkInit,check_init,g' \
  -e 's,checkNetwork,check_network,g' \
  -i support/testing/tests/init/*.py

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:58:16 +02:00
Ricardo Martincoski
ef8d1f1b15 support/testing: fix code style
Fix the trivial warnings from flake8:
 - remove modules imported but unused;
 - use 2 lines before class or module level method;
 - remove blank line at end of file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:56:49 +02:00
Ricardo Martincoski
2927f412be support/testing: standardize defconfig fragments style
Change all defconfig fragments to take advantage of
"cf3cd4388a support/tests: allow properly indented config fragment".

Make each defconfig fragment:
 - start after a backslash;
 - be declared as a multi-line string literal;
 - be indented one level more than the variable that contains it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:56:06 +02:00
Ricardo Martincoski
4a1d220d9e support/testing: allow to indent ccache defconfig fragment
Move the strip out of leading spaces in defconfig fragments from the
BRTest class to the Builder class. It actually postpones the strip out,
consequentially allowing test cases to post-process the defconfig in
their own __init__ before calling the __init__ method from BRTest.

Moving this code to the Builder class also allows any new test class
that inherits from BRTest to reuse the same code even if the new class
overrides the setUp method.

At same time, prepend a newline to the jlevel handling otherwise it
would stop working for defconfig fragments that don't end in a newline.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-10-06 18:53:13 +02:00
Yann E. MORIN
144dc9ca80 boot/grub: remove
grub is no longer maintained: it is stuck at version 0.97 with huge
patches that have no opportunity to be applied upstream, as upstream
has even renamed it grub-legacy.

Besides, it no longer builds correctly with recent binutils versions,
and even the huge patches we could grab from Debian do not help the
slightest.

Since upstream really considers it dead, and there are at least two
alternatives (grub2 and syslinux), just remove grub.

Add a legacy entry.

Remove the test cases as well.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-09-23 20:29:37 +02:00
Yann E. MORIN
ebe61135c9 support/tests: use relative test_dir
When we list the available tests, we use test_dir, which is set from
the path of the script. However, when we run the tests, we use the
hard-coded path.

Ditto to find the config file.

For consistency, always use test_dir.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-09-05 21:09:06 +02:00
Ricardo Martincoski
6e45e33f27 support/testing: allow to use a multiplier for timeouts
Add a parameter to run-tests to act as a multiplier for all timeouts of
emulator.
It can be used to avoid sporadic failures on slow host machines as well
in elastic runners on the cloud.

Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: rename multiplier to timeout_multiplier everywhere]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-08-10 10:08:06 +02:00
Yann E. MORIN
dd9dfb64ea support/testing: add runtime testing for read-only systemd
We add the 3 following combinations:

  - basic systemd, read-only, network w/ ifupdown
  - basic systemd, read-only, network w/ networkd
  - full systemd, read-only, network w/ networkd

The tests just verify what the /sbin/init binary is, and that we were
able to grab an IP address. More tests can be added later, for example
to check each systemd features (journal, tmpfiles...)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Arnout: regenerate .gitlab-ci.yml]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-02 21:00:44 +02:00
Yann E. MORIN
117835d5fc support/testing: add runtime testing for init systems
The "builtin" kernel does not boot a systemd-based system, so
we resort to building the same one as currently used by our
qemu_arm_vexpress_defconfig.

We test the 8 following combinations:

  - busybox, read-only, without network
  - busybox, read-only, with network
  - busybox, read-write, without network
  - busybox, read-write, with network

  - basic systemd, read-write, network w/ ifupdown
  - basic systemd, read-write, network w/ networkd
  - full systemd, read-write, network w/ networkd

  - no init system, read-only, without network

The tests just verify what the /sbin/init binary is, and that we were
able to grab an IP address. More tests can be added later, for example
to check each systemd features (journal, tmpfiles...)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: update .gitlab-ci.yml]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-02 17:42:49 +02:00
Ricardo Martincoski
d8ab6b2b54 testing/infra/emulator: remove qemu warnings about audio
The default audio backend for qemu is configured at compile time. It
generates annoying warning messages to qemu's stderr when running our
tests, like these:
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument

Explicitly set the audio backend to "none" at runtime to remove those
messages from our logs. There is no command line argument for this, so
use an environment variable when starting qemu.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-31 21:26:06 +02:00
Arnout Vandecappelle
6355ec2768 support/testing: add test of BR2_CCACHE with an external toolchain
We reuse TestExternalToolchainBuildrootuClibc and add ccache to its
configuration.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 18:35:15 +02:00
Ricardo Martincoski
b6bbb0782a testing/infra/basetest: move jlevel logic to constructor
As suggested by Arnout in [1].

While at it, simplify the logic by always appending the BR2_JLEVEL and
defaulting to 0 (the value copied from Config.in is used for 5 years now
and is very unlikely to change).

[1] http://patchwork.ozlabs.org/patch/790525/

Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:51:06 +02:00
Ricardo Martincoski
78c8958619 support/testing: fix run-tests -j
Since commit cf3cd4388a the -j option is
silently ignored.

The configuration lines are processed using '\n'.join().
This function adds intervening occurrences of the separator, but the
resulting string does not end at a separator.
 >>> "n".join(["a","b"])
 'anb'
It results in a defconfig that does not end in a newline.

When BR2_JLEVEL is added by -j logic to the defconfig it ends up
concatenated to the last line of the defconfig.
 BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=yBR2_JLEVEL=7
The resulting .config has the default BR2_JLEVEL=0.

Instead of just workaround this problem by adding a newline before
BR2_JLEVEL when -j is used, make the defconfig to end in a newline since
it is a more future-proof solution.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:50:52 +02:00
Ricardo Martincoski
046f968775 testing/infra/builder: dump defconfig to log
The defconfig is composed on-the-fly by test infra + tests.

Dump it to the logfile before running 'make olddefconfig' so it can
easily analysed when debugging.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-24 17:50:45 +02:00
Andrey Smirnov
d2fe2686d7 testing/tests/package: add basic unit test for IPython
Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
[Thomas: update .gitlab-ci.yml.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:55:05 +02:00
Andrey Smirnov
8559c55ed3 testing/tests/package/test_python: allow to change timeout
Depending on Python implementation used for testing, time it takes to
perform a given test can vary pretty significantly. To accout for that
allow individual test functions to specify different timeout value.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:54:37 +02:00
Andrey Smirnov
ded5fa3059 testing/tests/package/test_python: do not hardcode interpreter name
In order to be able to leverage the same test code for testing
different python interpreters (or wrappers around CPython) allow child
classes of TestPythonBase to override the name of the executable used
to run tests.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:53:27 +02:00
Andrey Smirnov
97278ebddd testing/tests/package/test_python: add TestPython3
Add Python3 version of TestPython2 to make sure both versions of
Python get unit-tested.

Modify the code of libc_time_test() to support that change (convert
the code to use Python3-style "print").

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
[Thomas: update .gitlab-ci.yml.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:52:42 +02:00
Andrey Smirnov
4c85d3e459 testing/tests/package/test_python: refactor TestPythonBase
Convert TestPythonBase to a true base class that only provides code
implementing various tests without defining tests themselves in a
"discoverable" form.

To retain correct testing functionality, add TestPython2 derived class
that uses code from TestPythonBase to define actual runnable test.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: fix typo in commit log, update .gitlab-ci.yml, both pointed
by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:38:21 +02:00
Andrey Smirnov
255b8ab406 testing/infra/emulator: allow to specify pexpect timeout
Some commands take more than 5 seconds to complete under QEMU, so add
provisions to allow individual unit-test to specify different duration
to avoid false negative test failures.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 22:37:06 +02:00
Yann E. MORIN
cf3cd4388a support/tests: allow properly indented config fragment
Currently, defining a config fragment in the runtime test infra requires
that the fragment not to be indented. This is beark, and causes grievance
when looking at the code (e.g. to fix it).

Just strip out all leading spaces/tabs when writing the configuration
lines into the config file, allowing in-line indented config fragments,
like so:

    class TestFoo(bla):
        config = bla.config + \
            """
            FOO=y
            # BAR is not set
            """

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-18 23:46:09 +02:00
Yann E. MORIN
3268968ccc support/tests: fix ext4 runtime test
The current test fails because of a legacy option, renamed during the
recent ext overhaul.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-14 19:37:14 +02:00
Arnout Vandecappelle
704db1586c support/testing: unbreak run-tests -l
In commit b78b50465c, the initialisation
of BRTest.builddir was moved to the __init__ function. However, it is
set based on BRTest.outputdir and that is only set when the -o argument
is given to run-tests. When called as "run-tests -l", there is no -o
argument so BRTest.outputdir remains unset.

To fix, keep BRTest.builddir at None when BRTest.outputdir is None.

While we're at it, drop the direct access to the class member. If a
subclass wishes to set outputdir to something else before calling
BRTest.__init__, they are free to do so.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 23:51:33 +02:00
Arnout Vandecappelle
b78b50465c support/testing: move BRTest initialisation to __init__
BRTest's setUp() method contains a few assignments that initialize its
member variables. Since we will want to use these in test case
overrides, move them to the __init__ function.

Also allow the config member to be overridden, rather than always
taking the class member.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 17:46:25 +02:00
Arnout Vandecappelle
73e7bcdd87 support/testing: strip /usr/ part from HOST_DIR
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 17:46:22 +02:00
Arnout Vandecappelle
8367d388d4 support/testing: remove references to host/usr
The tools are now installed in host/bin instead of host/usr/bin.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 16:53:56 +02:00