Commit Graph

1215 Commits

Author SHA1 Message Date
Peter Korsgaard
436daadc42 support/testing: drop explicit docker kernel options from docker test
Since commit 0390777bfa (package/docker-engine: needs some kernel
options), docker-engine now automatically ensures the needed kernel options
are enabled, so drop the explicit options from the kernel config.

23:19:27 TestDockerCompose                        Starting
23:19:28 TestDockerCompose                        Building
00:14:41 TestDockerCompose                        Building done
00:15:30 TestDockerCompose                        Cleaning up
.
----------------------------------------------------------------------
Ran 1 test in 3362.784s

OK

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-31 09:13:25 +02:00
Peter Korsgaard
4726cf9517 support/testing: drop explicit CGROUPFS_MOUNT from docker test
Since commit 4f8229653 (package/docker-engine: needs more runtime
dependencies), docker-engine now automatically pulls in cgroupfs-mount, so
drop the explicit handling of it in TestDockerCompose.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-31 09:13:21 +02:00
Thomas Petazzoni
98c99556e3 Makefile: properly account for custom tags in BR2_VERSION_FULL
BR2_VERSION_FULL is currently defined as follows:

  BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)

This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.

The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.

If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).

This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.

The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.

Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:

$ make print-version
2020.08-git-00758-gc351877a6e

So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:

$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e

This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.

So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.

In the above situation, with this commit applied, we get:

$ make print-version
dummy-tag-6-g6258cdddeb

(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)

Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-27 18:46:31 +02:00
Romain Naour
2ed3b0a586 gitlab-ci: check all defconfigs on every push
The defconfig check has been introduced by the previous
patch before the building each defconfig but those builds
are done every week or more.

Checking if a defconfig is valid can be done on every
push in the repository since it take few seconds.

This would allow to detect as soon as possible a problem
in a defconfig and eventually avoid breaking the build
while build testing all defconfig.

Introduce a new job template ".defconfig_check" in
gitlab-ci.yml.in and modify the generate-gitlab-ci-yml
to create a job for each defconfig to run the test.

Although, we could have used only one job to do all
tests, using one job per defconfig allow to identify
easily in gitlab which defconfig is falling.

Tested:
https://gitlab.com/kubu93/buildroot/pipelines/138331069
https://gitlab.com/kubu93/buildroot/pipelines/171223758

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 18:13:45 +02:00
Romain Naour
52b8ebc088 gitlab-ci: check generated config files
Use the script added by the previous patch to check
generated config files.

Tested on gitlab:
https://gitlab.com/kubu93/buildroot/pipelines/137597966

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 18:13:44 +02:00
Romain Naour
7385ccab18 support/scripts: add check-dotconfig.py
For the same reason as for 50b747f212,
we need to check if the generated configuration file (.config)
contains all symbols present in the defconfig file.

If not there is an issue with the defconfig.

This script will be used in .gitlab-ci.yml.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr:
  - strip defconfig lines when reading them
  - use a generator to read the defconfig lines
  - no need to strip() again when building the missing list
  - testing the list directly, not its len()
  - simply sys.exit(1) in the error condition
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 18:13:19 +02:00
Mark Corbin
468f728dcb support/config-fragments: use external riscv64 musl toolchain
Move to an external RISC-V 64 bit musl toolchain to ease the load
on the autobuilders.

Signed-off-by: Mark Corbin <mark@dibsco.co.uk>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-27 18:00:17 +02:00
Mark Corbin
c79a4d8772 support/config-fragments: add RISC-V 64-bit uclibc autobuild config
Add a RISC-V 64-bit autobuild configuration for the internal
toolchain with uclibc.

Signed-off-by: Mark Corbin <mark@dibsco.co.uk>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-27 18:00:17 +02:00
Romain Naour
93a2870966 gitlab: generate the gitlab-ci configuration before each build
Since Gitlab 12.9, Gitlab allow to trigger child pipeline with generated configuration file.
See: https://gitlab.com/gitlab-org/gitlab/-/issues/35632

This allow us to stop updating the .gitlab-ci.yml file when a
new defconfig is added to Buildroot.

Remove check-gitlab-ci.yml job since it is now uneeded.
Remove .gitlab-ci.yml make target.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[ann.morin.1998@free.fr: manual: no longer needed to update at all]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 13:45:52 +02:00
Eric Le Bihan
d7c944b2ae package/cargo: drop package
As no package depends on the standalone cargo package, it can be
dropped.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: David Pierret <david.pierret@smile.fr>
Tested-by: David Pierret <david.pierret@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-27 13:31:00 +02:00
James Hilliard
5bb1827111 package/meson: bump to version 0.55.0
Drop fix rpath patch which is no longer needed.

Drop g-ir-scanner/g-ir-compiler override patch which is now upstream.

Rebase remaining patches.

Meson now requires single quotes for cross-compilation.conf, replace
double quotes with single quotes.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-18 15:53:43 +02:00
Thomas Petazzoni
204d03ae43 support/scripts/pkg-stats: fix flake8 warning
This fixes the following flake8 warning:

support/scripts/pkg-stats:1005:9: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:23:13 +02:00
Gregory CLEMENT
7d2779ecbb support/script/pkg-stats: handle exception when version comparison fails
With python 3, when a package has a version number x-y-z instead of
x.y.z, then the version returned by LooseVersion can't be compared
which raises a TypeError exception:

Traceback (most recent call last):
  File "./support/scripts/pkg-stats", line 1062, in <module>
    __main__()
  File "./support/scripts/pkg-stats", line 1051, in __main__
    check_package_cves(args.nvd_path, {p.name: p for p in packages})
  File "./support/scripts/pkg-stats", line 613, in check_package_cves
    if pkg_name in packages and cve.affects(packages[pkg_name]):
  File "./support/scripts/pkg-stats", line 386, in affects
    return pkg_version <= cve_affected_version
  File "/usr/lib64/python3.8/distutils/version.py", line 58, in __le__
    c = self._cmp(other)
  File "/usr/lib64/python3.8/distutils/version.py", line 337, in _cmp
    if self.version < other.version:
TypeError: '<' not supported between instances of 'str' and 'int'

This patch handles this exception by adding a new return value when
the comparison can't be done. The code is adjusted to take of this
change. For now, a return value of CVE_UNKNOWN is handled the same way
as a CVE_DOESNT_AFFECT return value, but this can be improved later
on.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:22:55 +02:00
Kamel Bouhara
e068f33700 support/dependencies: add BR2_NEEDS_HOST_GCC_PLUGIN_SUPPORT
Some packages requires support on the build machine to create gcc
plugins. This commit adds a blind option,
BR2_NEEDS_HOST_GCC_PLUGIN_SUPPORT, which such packages can
select. When this option is enabled, the logic in support/dependencies
verifies that everything needed on the build machine to build gcc
plugins is available.

Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 11:20:27 +02:00
Thomas Petazzoni
45b174c8ad support/scripts/pkg-stats: remove debug cruft
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 23:30:52 +02:00
Peter Seiderer
0d0af210f7 support/testing: add python-gpiozero test
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 21:45:00 +02:00
Peter Seiderer
dc5761beb0 support/testing: add python-colorzero test
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 21:45:00 +02:00
Peter Seiderer
185398f619 package/python-colorzero: new package
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 21:45:00 +02:00
Adam Duskett
e9a02417f3 package/openjdk-bin: install to host/usr/lib/jvm
Buildroot currently installs openjdk-bin to $(HOST_DIR)/ instead of the more
traditional (for java installations) $(HOST_DIR)/usr/lib/jvm.

As described in https://bugs.busybox.net/show_bug.cgi?id=13001

"Openjdk-bin provides it's own libfreetype.so and places it into
$(HOST_DIR)/lib/. This library causes build failures with the
host-xapp_mkfontscale package due to the overwritten libfreetype.so.

mkfontscale.o: In function `doDirectory':
mkfontscale.c:(.text+0x1a80): undefined reference to `FT_Get_BDF_Property'
collect2: error: ld returned 1 exit status

Reproducing the error is done by repeating the following steps.
make host-freetype
make host-openjdk-bin
make host-xapp_mkfontscale"

There are two options for fixing this problem:

 1) add host-freetype and host-lksctp-tools as dependencies to host-openjdk-bin
    and then remove the provided libfreetype.so and libsctp.so libraries
    in a post_extract_hook.

 2) change the installation directory from $(HOST_DIR)/ to
    $(HOST_DIR)/usr/lib/jvm just like the target OpenJDK package and
    copy the entire source directories contents to the above location.

The second option provides the following advantages:
  - the directory structure is consistent with how we handle the target OpenJDK.

  - the HOST_OPENJDK_BIN_INSTALL_CMDS step is simplified.

  - packages such as Maven require directories of which we are currently not
    copying. These missing directories cause programs such as Maven to crash
    when running with an error such as
    "Can't read cryptographic policy directory: unlimited."

  - does not miss any other libraries that solution 1 would not cope with
    (e.g. libzip.so from host-libzip, or libnet.so from not-yet existing
    host-libnet, or libsctp.so from not-yet existing host-lksctp-tools)

Because the second option is both simple, easier to implement, is low-impact,
and fixes the problems described above wholly, it is the best to implement.

To implement the above changes, we must also modify the following files in the
same patch to match the host's new directory paths:

 - openjdk.mk
 - openjdk-jni-test.mk
 - openjdk-hello-world.mk

To avoid having to change all those packages in the future, expose two
new variables, HOST_OPENJDK_BIN_ROOT_DIR which contains the path where
the openjdk-bin was installed in, and JAVAC, which contains the path to
the javac compiler (modeled after the way the autoconf et al. variables
are set and exposed).

Tested with:
./support/testing/run-tests -o out -d dl tests.package.test_openjdk.TestOpenJdk

Fixes: https://bugs.busybox.net/show_bug.cgi?id=13001

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[yann.morin.1998@free.fr:
  - introduce HOST_OPENJDK_BIN_ROOT_DIR and JAVAC
  - expand and tweak the commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-06-18 11:01:41 +02:00
Yann E. MORIN
8fe9894f65 suport/download: fix git wrapper with submodules on older git versions
Older versions of git store the absolute path of the submodules'
repository as stored in the super-project, e.g.:

    $ cat some-submodule/.git
    gitdir: /path/to/super-project/.git/modules/some-submodule

Obviously, this is not very reproducible.

More recent versions of git, however, store relative paths, which
de-facto makes it reproducible.

Fix older versions by replacing the absolute paths with relative ones.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-06-17 22:58:22 +02:00
Yann E. MORIN
c62e78a85b core/br2-external: report better error messages
The error is misleading: it reports that no name was provided,
when in fact the external.desc file is missing.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>p
Reviewed-by: Romain Naour <romain.naour@gmail.com>
2020-06-15 11:01:54 +02:00
Yann E. MORIN
0ac7dcb73e core/br2-external: fix reporting errors
When a br2-external tree has an issue, e.g. a missing file, or does not
have a name, or the name uses invalid chars, we report that condition by
setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external,
which outputs it on stdout, and checked by the Makefile.

Before d027cd75d0, stdout was explicitly redirected to the generated
.mk file, with   exec >"${ofile}"   as the Makefile and Kconfig
fragments were generated each with their own call to the script, and
the validation phase would emit the BR2_EXTERNAL_ERROR variable in the
Makefile fragment.

But with d027cd75d0, both the Makefile and Kconfig fragments were now
generated with a single call to the script, and as such the semantics of
the scripts changed, and only each of the actual generators, do_mk and
do_kconfig, had their out put redirected. Which left do_validate with
the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

In turn, the stdout of the script would be interpreted by as part of the
Makefile. But this does not end up very well when a br2-external tree
indeed has an error:

  - missing a external.desc file:

    Makefile:184: *** multiple target patterns.  Stop.

  - empty external.desc file:

    Config.in:22: can't open file "output/.br2-external.in.paths"

So we must redirect the output of the validation step to the
Makefile fragment, so that the error message is correctly caught by the
top-level Makefile.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no
error, and thus the fragment is empty.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
2020-06-15 11:01:18 +02:00
Titouan Christophe
cc267d7ab4 support/testing: add runtime test for Redis
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-06-06 00:17:40 +02:00
Yann E. MORIN
7715be35d0 support/gnuconfig: update to 2020-04-26
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-05-09 14:18:49 +02:00
Yann E. MORIN
ca7a6ef8aa support/gnuconfig: add update script
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-05-09 14:18:39 +02:00
Yann E. MORIN
f45dc70833 support/gnuconfig: reference the correct sha1 we're using
In commit ff9f778c66 (support/gnuconfig: update to 2019-05-28), we
forgot to update the README to reference the sha1 we're using, keeping
the old one from 2016...

Update it now.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-05-09 14:18:33 +02:00
Francois Perrad
a3b3571fab support/testing: test_lua_utf8: fix test after bump to lua-utf8 0.1.2
The commit 0f0649140f introduced a change
in the module name: utf8 -> lua-utf8.

The packages-file-list.txt show the change in the files intalled on the rootfs:

[lua-utf8 0.1.2-2 latest version]
lua-utf8,./usr/lib/luarocks/rocks-5.1/luautf8/0.1.2-2/doc/LICENSE
lua-utf8,./usr/lib/luarocks/rocks-5.1/luautf8/0.1.2-2/doc/README.md
lua-utf8,./usr/lib/luarocks/rocks-5.1/luautf8/0.1.2-2/luautf8-0.1.2-2.rockspec
lua-utf8,./usr/lib/luarocks/rocks-5.1/luautf8/0.1.2-2/rock_manifest
lua-utf8,./usr/lib/lua/5.1/lua-utf8.so

[lua-utf8 1.2-0 previous version]
lua-utf8,./usr/lib/luarocks/rocks-5.1/utf8/1.2-0/doc/README.md
lua-utf8,./usr/lib/luarocks/rocks-5.1/utf8/1.2-0/utf8-1.2-0.rockspec
lua-utf8,./usr/lib/luarocks/rocks-5.1/utf8/1.2-0/rock_manifest
lua-utf8,./usr/lib/lua/5.1/utf8.so

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/526036036

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-27 22:03:21 +02:00
Romain Naour
e79018544d support/scripts/boot-qemu-image.py: wait before using expect
As reported by a gitlab runtime test [1] and on the mailing list
[2], some runtime tests are failing on slow host machines when
the qemu-system-<arch> is missing on the host.

The boot-qemu-image.py script need to wait some time after
calling pexpect.spawn() in order to make sure that the qemu
process has been executed in start-qemu.sh.

If start-qemu.sh failed due to missing qemu-system binary
an exception will be thrown by child.expect() and should be
catched by the error handling (pexpect.EOF).

After spending a lot of time to investigate with Yann E. MORIN
[3]. It seems that short-lived child processes are a corner-case
that is not very correctly handled...

Without adding a sleep(1), child.expect() can trigger an
exception before setting the exitstatus of the spawned
process. This issue can be reproduced on a gitlab runner or
by adding "exit 1" in the first line of start-qemu.sh
(after the shebang).

There is even the same workaround in some pexpect examples [4].

Thanks to Yann for the help while investigating the issue.

Tested:
https://gitlab.com/kubu93/buildroot/pipelines/138472925

[1] https://gitlab.com/kubu93/buildroot/pipelines/135487475
[2] http://lists.busybox.net/pipermail/buildroot/2020-April/280037.html
[3] http://patchwork.ozlabs.org/project/buildroot/patch/20200418161023.1221799-1-romain.naour@gmail.com/
[4] https://github.com/pexpect/pexpect/blob/master/examples/ssh_tunnel.py#L80

Fixes:
https://gitlab.com/kubu93/buildroot/-/jobs/509053135

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr: reorder imports]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-04-22 22:04:47 +02:00
Jugurtha BELKALEM
0c79350638 support/scripts/boot-qemu-image.py: boot Qemu images with Qemu-system.
This script is intended to be used by gitlab CI to test at runtime Qemu
images generated by Buildroot's Qemu defconfigs.

This allows to troubleshoot different issues that may be associated with
defective builds by lanching a qemu machine, sending root password,
waiting for login shell and then perform a shutdown.

This script is inspired by toolchain builder [1] and the Buildroot
testing infrastructure.

The gitlab CI will call this script for each defconfig build but only
Qemu defconfig will be runtime tested, all others defconfig are ignored.

Some Qemu defconfig must be used with a specific Qemu version (fork)
that is not always available, so the script doesn't error out when it
can't spawn a missing command. That condition is anyway printed in the
log.

Finally, the script start Qemu like it's done for the Buildroot
testing infrastructure (using pexpect).

Note:
We noticed some timeout issues with pexpect when the Qemu machine is
powered off. That's because Qemu process doesn't stop even if the
system is halted (after "System halted"). So the script doesn't error
out when such timeout occure. The behaviour depends on the architecture
emulated by Qemu.

[1] https://github.com/bootlin/toolchains-builder/blob/master/build.sh

Signed-off-by: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-04-13 21:51:13 +02:00
Heiko Thiery
f41056ec4b support/scripts/pkg-stats: add tilde '~' expansion for pathes
When the 'nvd-path', 'json' and 'html' are used like this:

  --html ~/foo

then the tilde expansion is properly done by the shell. However, when
they are used like this:

  --html=~/foo

The shell doesn't do the tilde expansion, and pkg-stats doesn't do
it. This commit modifies pkg-stats to ensure that tilde expansion is
done when parsing the 'nvd-path', 'json' and 'html' arguments.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: improve commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-12 14:49:45 +02:00
Romain Naour
3e6b97e09e support/testing: test_python_django: use the timeout_multiplier value
As reported by our gitlab runtime test [1] and on the mailing list
[2], the test_python_django is failing due to django server taking a
lot of time to start. Since the django server is started in background
through pexpect, we can't easily wait for the last startup line:

"January 01, 1970 - 00:00:41

Django version 3.0.4, using settings 'testsite.settings'

Starting development server at http://0.0.0.0:1234/

Quit the server with CONTROL-C."

In the failing gitlab job, we don't see such lines.

If we increase a lot the timout, the test passes.

Use timeout_multiplier introduced by [3] in order to adjust the
timeout.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/488816222

Tested:
https://gitlab.com/kubu93/buildroot/-/jobs/507458355

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/488816222
[2] http://lists.busybox.net/pipermail/buildroot/2020-April/279598.html
[3] 6e45e33f27

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-12 12:08:51 +02:00
James Hilliard
b7d251293a package/python-txaio: drop python 2 support
As of upstream commit 735eb608637e7bbab4082a541ac802cc919fec22,
available since version v20.1.1, support for Python 2.x has been
dropped, and Python >= 3.5 is required. So we make python-txaio depend
on python3, and remove the python2 test of python-txaio.

We also remove the python-six dependency which is no longer used by
txaio, since upstream commit 62b0e7eaa22769687df1de8f57374cb0a42bdc4d.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-11 14:48:45 +02:00
James Hilliard
8a99ce44b4 package/python-autobahn: drop python 2 support
Upstream commit
c0223223f8,
which has been part of the releases since v20.1.2 has dropped support
for Python 2.x. So python-autobahn is now only available for Python
3.x.

We therefore remove the Python 2.x autobahn tests, and while at it,
drop the python-six dependency, which is no longer needed since
upstream commit
79bd2ba41b,
part of the releases since v20.1.3.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-08 22:55:07 +02:00
Romain Naour
50b747f212 support/testing: check if the defconfig provided for testing is valid
Currently, the build continue even if some symbols disapear from
the generated dot config file (.config).

This patch add a new check in order to stop the test if one
of the provided symbol is missing. This must be treated as error.

For example, if a symbol disapear due to new dependency constraints.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
8e217262a8 support/testing: don't use TestPythonPackageBase.config and linaro toolchain
TestPythonPackageBase.config provide already the basic uClibc toolchain.
So by adding the symbols for the linaro toolchain some warning are printed
at while loading the configuration:

.config:16:warning: override: reassigning to symbol BR2_TOOLCHAIN_EXTERNAL
.config:16:warning: override: BR2_TOOLCHAIN_EXTERNAL changes choice state
.config:17:warning: override: reassigning to symbol BR2_TARGET_ROOTFS_CPIO
.config:21:warning: override: BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM changes choice state

So, some symbol disapear from the generated dot config (.config) leading
to an error due to a new check in the testsuite infra.

Since this test should use the Linaro toolchain, remove
TestPythonPackageBase.config add BR2_arm=y and disable the rootfs tar option.

While at it, re-order the options so that they appear in the same
order as they appear in a defconfig.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Adam Duskett <Aduskett@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
abe170d94f support/testing: test_lxc: use ARM arm external toolchain
Until now, the lxc test was using the ARM CodeSourcery 2014.05 armv5 toolchain.
But the recent systemd version bump to 245 added a toolchain dependency
on systemd package due to build issues with gcc < 5.0.

Before [1] the lxc test was failing to build with the ARM CodeSourcery 2014.05
toolchain. After [1], the test is faling at runtime since the
"BR2_INIT_SYSTEMD=y" symbol disapear from the dot config (.config) due to
the new toolchain dependency.

Fix this by using the same toolchain as for the systemd tests [2]

[1] 2196ee25ff
[2] b3d979c0d1

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
96ee994443 support/testing: test_tmux: remove BR2_GENERATE_LOCALE
The BR2_GENERATE_LOCALE can only be used by uClibc based toolchains with the
internal toolchain backend [1].

The test_tmux is using a external uClibc toolchain, so the
"BR2_GENERATE_LOCALE="en_US.UTF-8"" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

[1] https://git.buildroot.net/buildroot/commit/?id=bd0ffe2206fbd32baf7f4a1dc5fde81cfad70462

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
3bba3bfc59 support/testing: test_glxinfo: remove BR2_TOOLCHAIN_EXTERNAL_LOCALE
The BR2_TOOLCHAIN_EXTERNAL_LOCALE is only defined for uClibc based
custom external toolchains.

The test_glxinfo is using a glibc toolchain, so the
"# BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

There is the same problem with:
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
9cc3b0c7de support/testing: test_syslinux: remove BR2_TOOLCHAIN_EXTERNAL_LOCALE
The BR2_TOOLCHAIN_EXTERNAL_LOCALE is only defined for uClibc based
custom external toolchains.

The test_syslinux is using a glibc toolchain, so the
"# BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

There is the same problem with:
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
6fa5278511 support/testing: test_syslinux: fix flake8 errors
Reported by gitlab:
https://gitlab.com/kubu93/buildroot/-/jobs/499374911

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-06 22:16:36 +02:00
Romain Naour
b3d979c0d1 support/testing: fix systemd test by using a more recent gcc
As reported on the mailing list [0], there is a build issue with systemd
245 when using gcc < 5.0, due to the following build issue:

    ../src/shared/gpt.c:7:9: error: initializer element is not constant
             { GPT_ROOT_X86,              "root-x86"              },

The pre-built external toolchain we have for armv5 (the default with
just BR2_arm=y) is a very old toolchain from CodeSourcery, which has a
gcc 4.8; we have no other pre-built toolchains for armv5, except by
using a custom one, like those from the Bootlin toolchain builder. But
using a custom toolchain is not nice, as we want our runtime test to
test nominal configurations.

So, switch the systemd tests to use a Cortex-A9, so that we can use the
ARM 2019.12 toolchain, and with VFP, so that it can boot in the qemu
vexpress machine we use for the test-cases.

Fixes:
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144403
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144405
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144406
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144408
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144410
    https://gitlab.com/buildroot.org/buildroot/-/jobs/498144412

[0] http://lists.busybox.net/pipermail/buildroot/2020-April/278931.html

Signed-off-by: Romain Naour <romain.naour@gmail.com>
[yann.morin.1998@free.fr:
  - just use cortex-a9_VFP, instead of using a bootlin toolchain
  - adapt the commit log accordingly
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-04-05 20:32:10 +02:00
Thomas Petazzoni
e9b393c676 support/testing: add build tests for the syslinux bootloader
This commit adds four new tests for the syslinux bootloader:

 - Building on x86, for legacy BIOS
 - Building on x86, for EFI BIOS
 - Building on x86-64, for legacy BIOS
 - Building on x86-64, for EFI BIOS

Runtime testing in Qemu would certainly be possible, but is left as a
future addition to these tests.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 23:07:36 +02:00
Romain Naour
0207a65323 support/testing: update basic toolchain to bootlin bleeding-edge 2018.11-1
Update the toolchain being used by the testsuite infra.

The new toolchain 2018.11-1 is based on gcc 8.2, uClibc-ng 1.0.30,
linux-headers 4.14 and binutils 2.31.1.
Enable BR2_TOOLCHAIN_HAS_THREADS_DEBUG that is now required.

The old toolchain 2017.05 is based on gcc 4.9, uClibc-ng 1.0.25,
linux-headers 3.10 and binutils 2.27.

Tested with gitlab
https://gitlab.com/kubu93/buildroot/pipelines/132376578

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 22:59:34 +02:00
Thomas Petazzoni
d797b7daef support/config-fragments/autobuild: update OpenRISC toolchain
Following commit eee96b0f0a that adds a
gcc patch for OpenRISC, the OpenRISC pre-built toolchain was
rebuilt. Let's use this new toolchain version for the autobuilders.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-04-02 23:42:24 +02:00
Yaroslav Syrytsia
35c5cf56d2 Makefile: make-4.3 now longer un-escapes \# in macros
make-4.3 shipped with a backward incompatible change in how sharp signs
are handled in macros. Previously, up to make 4.2, the sharp sign would
always start a comment, unless backslash-escaped, even in a macro or a
fucntion call.

Now, the sharp sign is no longer starting a comment when it appears
inside such a macro or function call. This behaviour was supposed to be
in force since 3.81, but was not; 4.3 fixed the code to match the doc.

As such, use of external toolchains is broken, as we use the sharp sign
in the copy_toolchain_sysroot macro, in shell variable expansion to
strip off any leading /: ${target\#/}.

Fix that by applying the workaround suggested in the release annoucement
[0], by using a variable to hold a sharp sign.

[0] https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html

Signed-off-by: Yaroslav Syrytsia <me@ys.lc>
[yann.morin.1998@free.fr:
  - move the SHARP_SIGN definition out of Makefile and into support/
  - expand the commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-31 21:37:45 +02:00
Adam Duskett
f41ee450e5 support/testing/tests/package/sample_gst1_python.py: fix flake8 errors
The older version of flake8 on the autobuilders does not support typehints.
As such, flake8 throws the following error when scanning sample_gst1_python:
E999 SyntaxError: invalid syntax

Remove the typehinting from on_message to fix this issue.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-29 15:50:07 +02:00
Adam Duskett
207d276d98 support/testing: fix test_gst1_python
The config is missing the following packages:
  - BR2_PACKAGE_GST1_PLUGINS_BASE
    This is needed for the videotestsrc plugin which provides
    videotestsrc.

  - BR2_PACKAGE_GST1_PLUGINS_BAD
    This is needed for the debugutils plugin which provides
    fakevideosink.

  - BR2_PACKAGE_GST1_PLUGINS_BASE_PLUGIN_VIDEOTESTSRC
    videotestsrcplugin used by the example pipeline.

  - BR2_PACKAGE_GST1_PLUGINS_BAD_PLUGIN_DEBUGUTILS
    fakevideosink plugin used by the example pipeline.

The sample was also amended to fix:
  - no call to main.

  - using autovideosink instead of fakevideosink. Using Fakevideoskink
    is preferred because its primary purpose is for debugging and sample
    pipelines. Autovideosink does not work because there is no video
    output device.

  - No function on_message. Without this function, the script fails
    because of the missing function.

  - The script sets the pipeline to Gst.State.EOS instead of
    Gst.State.NULL which results in a failed pipeline state. When the
    state is set to NULL, gstreamer automatically calls EOS. Manually
    setting the pipeline state to EOS results in the following error:

        Trying to dispose element sink, but it is in PLAYING instead of
        the NULL state. You need to explicitly set elements to the NULL
        state before dropping the final reference, to allow them to
        clean up. A refcounting bug may also cause this problem in the
        application or some element.

In addition, the default timeout is set too low and would result in a
test failure, this has been changed to 200 seconds.

These issues are now properly fixed, and the sample script passes.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[yann.morin.1998@free.fr:
  - meld the two patches together
  - add the print() in the on_demand callback
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-27 18:31:45 +01:00
Thomas De Schampheleire
a570f63dd4 Revert "package/perl-crypt-ssleay: new package"
This reverts commit 25033cfb86.

This perl package is deprecated and gives following recommendations while
building:

    *** THIS IS NOT AN ERROR, JUST A MESSAGE FOR YOUR INFORMATION ***

        Do you really need Crypt::SSLeay?

        Starting with version 6.02 of LWP, https support was unbundled into
        LWP::Protocol::https. This module specifies as one of its prerequisites
        IO::Socket::SSL which is automatically used by LWP::UserAgent unless
        this preference is overridden separately. IO::Socket::SSL is a more
        complete implementation, and, crucially, it allows hostname
        verification. Crypt::SSLeay does not support this. At this point,
        Crypt::SSLeay is maintained to support existing software that already
        depends on it.

        However, it is possible that your software does not really depend on
        Crypt::SSLeay, only on the ability of LWP::UserAgent class to
        communicate with sites over SSL/TLS.

        If are using version LWP 6.02 or later, and therefore have installed
        LWP::Protocol::https and its dependencies, and do not explicitly use
        Net::SSL before loading LWP::UserAgent, or override the default socket
        class, you are probably using IO::Socket::SSL and do not really need
        Crypt::SSLeay.

        Before installing Crypt::SSLeay, you may want to try specifying a
        dependency on LWP::Protocol::https.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-26 22:26:20 +01:00
Marcin Niestroj
803fc6ad51 support/testing: add netdata test
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 22:25:40 +01:00
Thomas Petazzoni
1214109cdc support/testing/tests/package/test_python_gobject.py: fix flake8 warning
Fixes:

   support/testing/tests/package/test_python_gobject.py:4:1: E302 expected 2 blank lines, found 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:54:29 +01:00
Thomas Petazzoni
4a180c34b7 support/testing/tests/package/test_gst1_python.py: fix flake8 warning
Fixes:

  support/testing/tests/package/test_gst1_python.py:29:1: W391 blank line at end of file

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:54:29 +01:00
Thomas Petazzoni
75371a4a12 support/testing/tests/package/sample_python_gobject.py: fix flake8 warnings
Fixes:

  support/testing/tests/package/sample_python_gobject.py:5:1: E302 expected 2 blank lines, found 1
  support/testing/tests/package/sample_python_gobject.py:8:7: E111 indentation is not a multiple of four
  support/testing/tests/package/sample_python_gobject.py:11:1: E305 expected 2 blank lines after class or function definition, found 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:54:29 +01:00
Thomas Petazzoni
15f354e7df support/testing/tests/package/sample_gst1_python.py: fix flake8 warnings
Fixes the following flake8 warnings:

  support/testing/tests/package/sample_gst1_python.py:5:1: F401 'time' imported but unused
  support/testing/tests/package/sample_gst1_python.py:7:1: E402 module level import not at top of file
  support/testing/tests/package/sample_gst1_python.py:21:12: W292 no newline at end of file

For the E402 warning, we add a "noqa" marker, as we really want the
gi.require_version() to be before.

There is still one remaining warning to be fixed:

  support/testing/tests/package/sample_gst1_python.py:18:28: F821 undefined name 'on_message'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:54:13 +01:00
Thomas Petazzoni
7e025dd80b support/testing/tests/init/init_openrc: fix flake8 F401 warning
Fixes:

support/testing/tests/init/test_openrc.py:1:1: F401 'infra.basetest' imported but unused

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:28:21 +01:00
Thomas Petazzoni
3b5bc480a5 support/scripts/pkg-stats: fix flake8 E722 warning
flake8 complains with:

  support/scripts/pkg-stats:339:13: E722 do not use bare 'except'

Due to the construct:

  try:
     something
  except:
     print("some message")
     raise

Which is in fact OK because the exception is re-raised. This issue is
discussed at https://github.com/PyCQA/pycodestyle/issues/703, and the
general agreement is that these "bare except" are OK, and should be
ignored from flake8 using a noqa statement.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
f7f33771b3 support/scripts/pkg-stats: fix flake8 E501 warning
Fixes:

support/scripts/pkg-stats:281:133: E501 line too long (139 > 132 characters)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
198d76efb3 support/scripts/pkg-stats: fix flake8 E117 warning
Fixes:

  support/scripts/pkg-stats:146:17: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
e03bdef0ec support/scripts/pkg-stats: fix flake8 E302 warning
Fixes:

  support/scripts/pkg-stats:57:1: E302 expected 2 blank lines, found 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
769f98c18c support/scripts/pkg-stats: fix flake8 E402 warning
flake8 complains with:

pkg-stats:38:1: E402 module level import not at top of file

This is due to sys.path.append() being before the import from
getdeveloperlib, but we really need this sys.path.append() to be
before, so let's ignore this flake8 warning.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:53 +01:00
Adam Duskett
1101d7cb09 support/testing/: change python-gobject to use prebuilt kernel
Currently, we build a complete AArch64 system, including a kernel. This
can take quite some time.

Switch to an armv7 system, which allows us to use one a prebuilt kernel,
thus significantly reducing the test time.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[yann.morin.1998@free.fr:
  - use a standalone config; don't inherit from TestPythonPackageBase
  - use the default external toolchain (ARM, not Linaro)
  - rewrite commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-22 15:25:34 +01:00
Adam Duskett
3f9bf62953 support/testing: add gst1-python test case
This test case runs a simple pipeline for 100 frames to ensure that
gst1-python works properly.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-22 11:55:31 +01:00
Thomas Petazzoni
152f8ff424 support/config-fragments: update Buildroot toolchains to 2020.02
This commit updates all our toolchain configuration fragments for
pre-built Buildroot toolchains to use toolchains built with Buildroot
2020.02.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-21 16:58:17 +01:00
Thomas Petazzoni
c3c4b3dfa8 support/scripts/check-kernel-headers.sh: do not print error for loose checks
The C program inside check-kernel-headers.sh has two checking mode: a
strict and a loose one.

In strict mode, we want the kernel headers version declared by the
user to match exactly the one of the toolchain.

In loose mode, we want the kernel headers version of the toolchain to
be greater than or equal to the one declared by the user: this is used
when we have a toolchain that has newer headers than the latest
version known by Buildroot.

However, in loose mode, we continue to show the "Incorrect kernel
headers version" message, even though we then return a zero error
code. This is very confusing: you see an error displayed on the
terminal, but the build goes on.

We fix that by first doing the loose check first, and returning 0 if
it succeeds. And then we move on with the strict check where we want
the version to be identical.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-21 15:47:03 +01:00
Francois Perrad
6500cb3cc6 package/lua-lyaml: new package
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-20 23:00:30 +01:00
Thomas De Schampheleire
0c839dc11c support/testing: add missing test file for perl-html-parser
This file was created by utils/scancpan while adding other packages but
apparently not yet added in the repo.

Assign this test case to Bernd in the DEVELOPERS file since he is
listed as the maintainer for this package.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-15 21:38:56 +01:00
Thomas De Schampheleire
25033cfb86 package/perl-crypt-ssleay: new package
Added via utils/scancpan, adding a host dependency to perl-try-tiny and
target dependency to openssl.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-15 21:38:56 +01:00
Thomas De Schampheleire
f8638b99ae package/perl-lwp-protocol-https: new package
Dependency for perl-crypt-ssleay (added in subsequent commit).

Added via utils/scancpan, without changes.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-15 21:38:56 +01:00
Adam Duskett
179e6766a6 package/python-gobject: bump version to 3.36.0
Other changes:
  - Convert the package to a meson package.
  - Remove 0001-add-PYTHON_INCLUDES-override.patch as it no longer applies.
  - Add gobject-introspection as a dependency.
  - Add the package under myself in the DEVELOPERS file.

Because gobject-introspection is now a dependency of python-gobject, the test
must be updated at the same time.

 - Change TestPythonPy2Gobject to TestPythonPy3Gobject as
   gobject-introspection requires python3.

 - Refactor test_python_gobject.py to no longer inherit the
   TestPythonPackageBase class, as this class uses a base config that does not
   support gobject-introspection.

 - Update sample_python_gobject to use Glib to find the path of sh.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-12 22:21:55 +01:00
Adam Duskett
23d3433302 support/testing: add openrc tests
Two simple tests to ensure that openrc boots without any services crashing
with a read only and a read write filing system.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Carlos Santos <unixmania@gmail.com>
[yann.morin.1998@free.fr: really check the init process]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-11 18:50:38 +01:00
Peter Korsgaard
fd99eb5016 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-09 15:17:09 +01:00
Peter Korsgaard
5f5477b0ab Update for 2020.02
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-08 22:23:03 +01:00
Heiko Thiery
759521dae6 support/scripts/pkg-stats: add list of status checks to the json output
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:38:26 +01:00
Heiko Thiery
fb879c1954 support/scripts/pkg-stats: set status to 'na' for virtual packages
If there is no infra set or infra is virtual the status is set to 'na'.

This is done for the follwing checks:
 - license
 - license-files
 - hash
 - hash-license
 - patches
 - version

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:37:23 +01:00
Heiko Thiery
8d77ecbad0 support/scripts/pkg-stats: add defconfig support
Scan configs directory and create Defconfig objects.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:37:02 +01:00
Heiko Thiery
d31fadfbf5 support/scripts/pkg-stats: store pkg dir path
This value can be used for later processing.

In the buildroot-stats application this is used to create links pointing
to the git repo of buildroot.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:54 +01:00
Heiko Thiery
0e267518cb support/scripts/pkg-stats: add package count to stats
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:45 +01:00
Heiko Thiery
f422fa991f support/scripts/pkg-stats: add package status
Unify the status check information. The status is stored in a tuple. The
first entry is the status that can be 'ok', 'warning' or 'error'. The
second entry is a verbose message.

The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check

With that status information the following variables are replaced:
has_license, has_license_files, has_hash, url_status

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:38 +01:00
Heiko Thiery
5b7278e5f1 support/scripts/pkg-stats: store licences of package
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:17 +01:00
Heiko Thiery
c1fc827934 support/scripts/pkg-stats: set developers info
Use the function 'parse_developers' function from getdeveloperlib that
collect the information about the developers and the files they
maintain. Then set the maintainer(s) to each package.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:08 +01:00
Heiko Thiery
b1916b0a8d support/scripts/pkg-stats: store patch files for the package
Remove the patch_count attribute and use a class property instead.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 15:59:52 +01:00
Heiko Thiery
c46e707182 support/scripts/pkg-stats: store latest version in a dict
This patch changes the type of the latest_version variable to a dict.
This is for better readability/usability of the data. With this the json
output is more descriptive in later processing of the json output.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 15:59:14 +01:00
Titouan Christophe
28adf09b89 support/scripts/pkg-stats: clear multiprocessing pools after use
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:59:08 +01:00
Titouan Christophe
fb05ab2242 support/scripts/pkg-stats: decode subprocess output for python3
In Python 3, the functions from the subprocess module return bytes
(and no longer strings as in Python 2), which must be decoded for
further text operations.

Now, pkg-stats can be run in Python 3.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:59:04 +01:00
Thomas Petazzoni
1097c0427d support/scripts/pkg-stats: properly ignore CVEs in <pkg>_IGNORE_CVES
It seems like throughout the series that the CVE pkg-stats support
went through, the support for ignoring CVEs in the per-package
<pkg>_IGNORE_CVES variable was forgotten.

Let's re-introduce this, which is now very simple thanks to the CVE
class, its .identifier() propertly and the .is_cve_ignored() method of
the Package class

Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:58:41 +01:00
Peter Korsgaard
22e833af5e Config.in: drop BR2_NEEDS_HOST_{JAVAC,JAR}
With classpath removed, no packages select these symbols any more - So drop
them and their corresponding logic in dependencies.sh / genrandconfig.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-03 23:55:48 +01:00
Titouan Christophe
54645c0b39 support/scripts/pkg-stats: clear multiprocessing pools after use
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-02 23:35:39 +01:00
Titouan Christophe
304b141a97 support/scripts/pkg-stats: decode subprocess output for python3
In Python 3, the functions from the subprocess module return bytes
(and no longer strings as in Python 2), which must be decoded for
further text operations.

Now, pkg-stats can be run in Python 3.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-02 23:35:26 +01:00
Romain Naour
b1e4404c04 support/testing: test_systemd.py: add linux fragment to enable CONFIG_BINFMT_MISC
While investigating [1] one units failed due to missing kernel option
CONFIG_BINFMT_MISC needed by "proc-sys-fs-binfmt_misc.mount" service.

It's because the kernel support autofs4 but not MISC binaries.

Since the systemd test infra use the default defconfig (vexpress),
we need to provide a linux fragment to enable CONFIG_BINFMT_MISC.

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/454255917

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr:
  - move the kernel config with the others in conf/
]
Tested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-01 18:43:42 +01:00
Titouan Christophe
a35f51cee1 support/scripts/pkg-stats: iterate over CVEs in streaming
The NVD files that are used to build the list of CVEs affecting
Buildroot packages are quite large (a few hundreds MB of json),
and cause the pkg-stats scripts to have a huge memory footprint
(a few GB with Python 2.7).

However, because we only need to iterate on CVE items one by one,
we can process them in streaming (ie decoding one CVE at a time
from the JSON representation). Because the json module from the
python standard library does not support such a mode of operation,
we switch to the third-party package ijson, which is compatible
with both Python 2 and Python3.

To run the script with these modifications, one should install
the ijson python package. This can be done with pip:
`pip install ijson`. On Debian based distributions, this can
also be done with the apt package manager:
`apt install python-ijson`.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-24 22:22:58 +01:00
Yegor Yefremov
593bba41ab support/testing: add libftdi1 test case
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-23 11:12:09 +01:00
Titouan Christophe
712f81c41c support/scripts/pkg-stats: iterate over CVEs in streaming
The NVD files that are used to build the list of CVEs affecting
Buildroot packages are quite large (a few hundreds MB of json),
and cause the pkg-stats scripts to have a huge memory footprint
(a few GB with Python 2.7).

However, because we only need to iterate on CVE items one by one,
we can process them in streaming (ie decoding one CVE at a time
from the JSON representation). Because the json module from the
python standard library does not support such a mode of operation,
we switch to the third-party package ijson, which is compatible
with both Python 2 and Python3.

To run the script with these modifications, one should install
the ijson python package. This can be done with pip:
`pip install ijson`. On Debian based distributions, this can
also be done with the apt package manager:
`apt install python-ijson`.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-20 21:31:05 +01:00
Thomas Petazzoni
60f2de1f12 support/scripts/pkg-stats: properly ignore CVEs in <pkg>_IGNORE_CVES
It seems like throughout the series that the CVE pkg-stats support
went through, the support for ignoring CVEs in the per-package
<pkg>_IGNORE_CVES variable was forgotten.

Let's re-introduce this, which is now very simple thanks to the CVE
class, its .identifier() propertly and the .is_cve_ignored() method of
the Package class

Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-19 08:22:09 +01:00
Yegor Yefremov
5abe7e4ce3 support/run-tests: reorder imports
Reorder imports using the isort utility to fix a warning from pylint3:

wrong-import-order: standard import "import multiprocessing" should be
placed before "import nose2"

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-17 10:13:08 +01:00
Yegor Yefremov
ea31dc1cd4 support/run-tests: check for empty sequences in a pythonic way
According to PEP8 empty sequences should be checked as booleans.

Fixes the following PEP8 warning:
Do not use `len(SEQUENCE)` to determine if a sequence is empty

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-16 15:38:03 +01:00
Thomas Petazzoni
4a157be9ef support/scripts/pkg-stats: add support for CVE reporting
This commit extends the pkg-stats script to grab information about the
CVEs affecting the Buildroot packages.

To do so, it downloads the NVD database from
https://nvd.nist.gov/vuln/data-feeds in JSON format, and processes the
JSON file to determine which of our packages is affected by which
CVE. The information is then displayed in both the HTML output and the
JSON output of pkg-stats.

To use this feature, you have to pass the new --nvd-path option,
pointing to a writable directory where pkg-stats will store the NVD
database. If the local database is less than 24 hours old, it will not
re-download it. If it is more than 24 hours old, it will re-download
only the files that have really been updated by upstream NVD.

Packages can use the newly introduced <pkg>_IGNORE_CVES variable to
tell pkg-stats that some CVEs should be ignored: it can be because a
patch we have is fixing the CVE, or because the CVE doesn't apply in
our case.

>From an implementation point of view:

 - A new class CVE implement most of the required functionalities:
   - Downloading the yearly NVD files
   - Reading and extracting relevant data from these files
   - Matching Packages against a CVE

 - The statistics are extended with the total number of CVEs, and the
   total number of packages that have at least one CVE pending.

 - The HTML output is extended with these new details. There are no
   changes to the code generating the JSON output because the existing
   code is smart enough to automatically expose the new information.

This development is a collective effort with Titouan Christophe
<titouan.christophe@railnova.eu> and Thomas De Schampheleire
<thomas.de_schampheleire@nokia.com>.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-15 16:49:07 +01:00
Romain Naour
faec5c583e support/testing/glxinfo: explicitely enable GLX
Since [1], the GLX support is enabled by BR2_PACKAGE_MESA3D_OPENGL_GLX
symbol.

Since [2], only one swrast provider can be built.
Keep BR2_PACKAGE_MESA3D_DRI_DRIVER_SWRAST.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/400391349

[1] 5cb821d563
[2] 09a0a28507

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-15 11:59:13 +01:00
Peter Korsgaard
5fd8dd203a toolchain: use consistent code style for C code
Most, but not all our C code follows the Linux kernel code style (as
documented in Documentation/process/coding-style.rst).  Adjust the few
places doing differently:

- Braces:
  ..but the preferred way, as shown to us by the prophets Kernighan
  and Ritchie, is to put the opening brace last on the line

- Spaces after keywords:
  Use a space after (most) keywords

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-08 22:10:06 +01:00
Vincent Fazio
338e62bd5d toolchain: allow using custom headers newer than latest known ones
When Buildroot is released, it knows up to a certain kernel header
version, and no later. However, it is possible that an external
toolchain will be used, that uses headers newer than the latest version
Buildroot knows about.

This may also happen when testing a development, an rc-class, or a newly
released kernel, either in an external toolchain, or with an internal
toolchain with custom headers (same-as-kernel, custom version, custom
git, custom tarball).

In the current state, Buildroot would refuse to use such toolchains,
because the test is for strict equality.

We'd like to make that situation possible, but we also want the user not
to be lenient at the same time, and select the right headers version
when it is known.

So, we add a new Kconfig blind option that the latest kernel headers
version selects. This options is then used to decide whether we do a
strict or loose check of the kernel headers.

Suggested-by: Aaron Sierra <asierra@xes-inc.com>
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
  - only do a loose check for the latest version
  - expand commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-08 20:25:10 +01:00
Titouan Christophe
1b2319f931 support/testing: add runtime test for Crudini (py2 and py3)
This also adds the new tests to the gitlab CI configuration.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-05 16:23:42 +01:00
Matt Weber
496a43c777 support/testing: new opkg test case
- Validates an archive can be installed and removed
- Builds an archives that uses postinst and prerm scripts

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-03 22:17:42 +01:00
Yegor Yefremov
96bbea75a3 support/testing: add python-can test case
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-01-31 08:05:21 +01:00
Luca Ceresoli
ec78068972 core/legal-info: update list of saved material in README
The README file saved by legal-info does not mention the host package
variant of the saved material. Add them.

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-18 18:38:42 +01:00
Yann E. MORIN
a95ea4b64d core/dependencies: check if we need to build our own host-coreutils
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-01-06 21:43:38 +01:00
Yann E. MORIN
ed56b7b547 support/testing: really fix python-gitdb2 test
The test-case for python-gitdb2 consists solely in verifying that the
module can indeed be imported.

However, flake8 errors out on unused imports. Furthermore, it also
errors about wildcard imports, as it can detect unused symbols.

Commit d8c86be9cd (support/testing: fix python-gitdb2 test) tried to
address this issue, by explicitly squelching the two errors, F401 and
F403.

While that works on recent distros, the image used by our docker
pipeline is laggign behind and the flake8 there only handles at most a
single error in the noqa list.

Do as is done with the other python samples, and just blindly ignore
all errors.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-04 09:29:36 +01:00
Yann E. MORIN
8dbea2216d support/config-fragments/autobuild: fix riscv toolchains
In commit aee39cbf27 (arch/riscv: set the default float ABI based on
ISA extensions), the default ABI changed, so the config fragments used
by the autobuilders were adapated accordingly, in commit f89871e810
(support/config-fragments: fix br-riscv{32,64} toolchain fragments).

But now, we need to revert again, because the newer toolchains are now
using the default ABI again.

We do not really do a revert, though, because the original change was
right, and a revert would mean it was not.

Fixes:
    http://autobuild.buildroot.org/results/b59/b593267fb9fc9a002b977e049b2a5389dbaded30/ (riscv32)
    http://autobuild.buildroot.org/results/b42/b42a4b22b29f47d5c85be119b310f1dfb61112a1/ (riscv64)
    ... and so many others on various packages...

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-01-02 09:15:43 +01:00
Thomas Petazzoni
2d67b66df9 support/config-fragments/autobuild: update Buildroot toolchains to 2019.11
Simple bump of the toolchain components. For nios2, the toolchain now
has SSP support as well.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - fix version it commit title
  - mention SSP for nios2
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-30 13:18:03 +01:00
Heiko Thiery
89f5e98932 support/download/svn: generate reproducible svn archives
To generate a reproducible archive from a svn repository mainly the same
aproach is done like for the archives from a git repository.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[yann.morin.1998@free.fr: get the date of the revision]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-30 10:50:21 +01:00
Titouan Christophe
9fa2add810 support/testing: add test for python-avro
This adds a test case for python-avro, with a script that
performs a simple deserialization.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-29 21:38:33 +01:00
Yann E. MORIN
d8c86be9cd support/testing: fix python-gitdb2 test
Cthe test-case for python-gitdb2 consists solely in verifying that the
module can indeed be imported.

However, flake8 errors out on unused imports. Furthermore, it also
errors about wildcard imports, as it can detect unused symbols.

Squelch those errors.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-18 21:12:22 +01:00
Thomas Petazzoni
2e42840e2a package/pkg-utils.mk: rework implementation of extractor-system-dependency
Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
we can use them to implement extractor-system-dependency: if for a
given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
variable is empty, then it means we need the corresponding extractor
tool to be provided by the system.

Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
longer used, so we can drop it from support/dependencies/.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-17 22:09:48 +01:00
Adam Duskett
5ef1679635 support/testing: add gitdb2 test
Add a simple test case that imports the module.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-16 23:36:45 +01:00
Adam Duskett
f81f673a78 support/testing: add smmap2 test
Add a simple test case that imports the module and instantiates a new
SlidingWindowMapManager class.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-16 23:28:08 +01:00
Thomas Petazzoni
fbee5c8e91 Revert "support/download/git: rename local refs to avoid confusing Git warning"
This reverts commit 6f35d96756.

Repeat after me: on the master branch you will not work. On the master
branch you will not work.

This definitely shouldn't have been pushed. Sorry about that.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-12 21:24:35 +01:00
Thomas Petazzoni
6f35d96756 support/download/git: rename local refs to avoid confusing Git warning
Running "git fetch origin ${cset}:${cset}" to create a local ref
${cset} from the remote ref ${cset} causes Git to issue a warning like
the below, when the version is a full commit hash:

===

warning: refname '49eb4ecb1ef9879ebc6789a1bdb536ab2b1d9871' is ambiguous.
Git normally never creates a ref that ends with 40 hex characters
because it will be ignored when you just specify 40-hex. These refs
may be created by mistake. For example,

  git switch -c $br $(git rev-parse ...)

where "$br" is somehow empty and a 40-hex ref is created. Please
examine these refs and maybe delete them. Turn this message off by
running "git config advice.objectNameWarning false"

===

This warning is very confusing for users, and is caused by the fact
that Git doesn't like our local ref name to look like a commit hash.

So, this commit proposes to fix the issue by having the local ref
named buildroot-${cset}, i.e
buildroot-${version-specified-by-the-package}.

The generated tarballs are exactly identical, nothing changes, it is
really just internally the local ref we are using to checkout the
correct version that is different. And it avoids the confusing
warning.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-12 15:42:50 +01:00
Thomas Petazzoni
97dee44a6c package/pkg-generic.mk, support/scripts/fix-rpath: fix per-package regexp
Commit c4e6d5c8be ("core: implement
per-package SDK and target") had a mistake on the regexp that is used
to match $(PER_PACKAGE_DIR)/<something>/, and due to this, the regexp
was never matched.

The + sign in [^/]+ which was suggested by Yann E. Morin during the
review of the per-package patch series (instead of [^/]*) needs to be
escaped to be taken into account correctly. Without this, the regexp
doesn't match, and the replacement is not done, causing:

 (1) For the libtool fixup in pkg-generic.mk, the lack of replacement
     causes libtool .la files to not be tweaked as expected, which it
     turn causes build failures reported by the autobuilder.

 (2) For the fix-rpath, the RPATH of host binaries in the SDK were not
     correct.

Interestingly, we have the same regexp in
support/scripts/check-host-rpath, but here the + sign does not need to
be escaped.

Fixes:

  http://autobuild.buildroot.net/results/d4d996f3923699e266afd40cc7180de0f7257d99/ (libsvg-cairo)
  http://autobuild.buildroot.net/results/56330f86872f67a2ce328e09b4c7b12aa835a432/ (bind)
  http://autobuild.buildroot.net/results/9e0fc42d2c9f856b92954b08019b83ce668ef289/ (ibrcommon)
  and probably a number of other similar issues

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-12 08:27:54 +01:00
Arnout Vandecappelle (Essensium/Mind)
17638bc3ad support/testing: test_python_django: add missing line
flake8 complains with:

test_python_django.py:25:1: E302 expected 2 blank lines, found 1

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-06 00:32:34 +01:00
Adam Duskett
8583b2c6d8 support/testing: add python-django test
This test comprises of four simple steps:
  1: Start a new simple project called testsite.
  2: Run ./manage.py migrate on the new testsite.
  3: Run ./manage.py runserver 0.0.0.0:1234 & sleep 30
    - The sleep 30 is necessary as it may take several seconds for
      the django server to fully start.
  4: Run netstat to ensure the server opened port 1234.

Signed-off-by: Adam Duskett <aduskett@greenlots.com>
[Thomas: use self.assertRunOk() when appropriate]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-05 23:02:14 +01:00
Adam Duskett
600132f8c0 support/testing: add python-gobject test
Add a simple test case that imports the module.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-03 23:49:34 +01:00
Peter Korsgaard
7b24bd59c8 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-02 09:39:41 +01:00
Peter Korsgaard
836b84a774 Update for 2019.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-01 22:39:47 +01:00
Thomas Petazzoni
c4e6d5c8be core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.

There are two main benefits:

 - Packages will now see only the dependencies they explicitly list in
   their <pkg>_DEPENDENCIES variable, and the recursive dependencies
   thereof.

 - We can support top-level parallel build properly, because a package
   only "sees" its own host directory and target directory, isolated
   from the build of other packages that can happen in parallel.

It works as follows:

 - A new output/per-package/ directory is created, which will contain
   one sub-directory per package, and inside it, a "host" directory
   and a "target" directory:

   output/per-package/busybox/target
   output/per-package/busybox/host
   output/per-package/host-fakeroot/target
   output/per-package/host-fakeroot/host

   This output/per-package/ directory is PER_PACKAGE_DIR.

 - The global TARGET_DIR and HOST_DIR variable now automatically point
   to the per-package directory when PKG is defined. So whenever a
   package references $(HOST_DIR) or $(TARGET_DIR) in its build
   process, it effectively references the per-package host/target
   directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
   is handled as well.

 - Of course, packages have dependencies, so those dependencies must
   be installed in the per-package host and target directories. To do
   so, we simply rsync (using hard links to save space and time) the
   host and target directories of the direct dependencies of the
   package to the current package host and target directories.

   We only need to take care of direct dependencies (and not
   recursively all dependencies), because we accumulate into those
   per-package host and target directories the files installed by the
   dependencies. Note that this only works because we make the
   assumption that one package does *not* overwrite files installed by
   another package.

   This is done for "extract dependencies" at the beginning of the
   extract step, and for "normal dependencies" at the beginning of the
   configure step.

This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:

 - The filesystem image creation code cannot work.

 - We don't have a SDK to build code outside of Buildroot.

In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.

This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:

 - The check-host-rpath script, which verifies at the end of each
   package installation that it has the appropriate RPATH, is modified
   to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
   a correct RPAT.

 - The fix-rpath script, which mungles the RPATH mainly for the SDK
   preparation, is modified to rewrite the RPATH to not point to
   per-package directories. Indeed the patchelf --make-rpath-relative
   call only works if the RPATH points to the ROOTDIR passed as
   argument, and this ROOTDIR is the global host directory. Rewriting
   the RPATH to not point to per-package host directories prior to
   this is an easy solution to this issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 14:24:05 +01:00
Pierre-Jean Texier
bda4008975 support/testing: basetest.py: fix code style
Fix these warnings:

W291 trailing whitespace

Fixes:
 - https://gitlab.com/buildroot.org/buildroot/-/jobs/360824861

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 21:40:52 +01:00
Pierre-Jean Texier
bd18e74eeb support/testing: test_lxc.py: fix code style
Fix these warnings:

E122 continuation line missing indentation or outdented
E127 continuation line over-indented for visual indent
E265 block comment should start with '# '
E302 expected 2 blank lines, found 1
F401 'pexpect' imported but unused

Fixes:
 - https://gitlab.com/buildroot.org/buildroot/-/jobs/360824861

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 21:40:41 +01:00
Patrick Havelange
1ca16b1feb support/testing: add lxc test
The test starts a simple container with an iperf3 server.
The container is using the tini init system, with a shared rootfs.
An iperf3 client is started from the host to check that the container
is really up and running.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-20 22:38:25 +01:00
Patrick Havelange
63966e56a3 support/testing: add assertRunOk method to BRTest class
This method asserts that the given command ran successfully.
The goal is for it to be used by the different tests when needed.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-20 22:34:53 +01:00
James Hilliard
d4773b610a support/dependencies: set cmake version min to 3.10
This is required by wpewebkit and webkitgtk.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Adrian Perez de Castro <aperez@igalia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 23:01:47 +01:00
Ricardo Martincoski
4a40d36f13 support/testing: switch to Python 3 only
Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.

In order to accomplish the change to Python 3:
 - change the shebang for run-tests;
 - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
 - when writing the downloaded binary files, explicitly open the output
   file as binary;
 - when subprocess is used to retrieve the text output from commands,
   explicitly ask for text output. For this, use 'universal_newlines'
   because 'text' was added only on Python 3.7;
 - when pexpect is used to retrieve the text output from qemu or git,
   explicitly ask for text output using 'encoding';
 - the code using csv currently follows the example in the documentation
   for the Python 2 module, change it to follow the example in the
   documentation for the Python 3 module;
 - fix the relative import for test_git.py to be Python 3 compliant.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 22:14:07 +01:00
Ricardo Martincoski
8b44338ec5 support/docker: add python3
The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2
and pexpect to the docker image used to run runtime tests.

Keep the Python 2 variant of those modules to allow a gradual
transition.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 20:24:10 +01:00
Carlos Santos
31d1fb27b0 support/scripts/genimage.sh: pass an empty rootpath to genimage
genimage makes a full copy of the given rootpath to ${GENIMAGE_TMP}/root
so passing TARGET_DIR would be a waste of time and disk space. We don't
rely on genimage to build the rootfs image, just to insert a pre-built
one in the disk image.

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 12:19:32 +01:00
Yann E. MORIN
5ee815ab6a docs/manual: bump our gcc/g++ requirements to 4.8
Currently, we only require a gcc 4.4 version, which now is pretty old
(released in April 2009).  This requirement is not even tested nowadays,
with our oldest autobuilder having a 4.7 version only.

And even then, 4.7 is still old enough that it prevents us from
upgrading some packages. For example cmake 3.10+ requires C++11
constructs that were only added in gcc 4.8 (when C++11 support was
finally completed in gcc).

So, update our requirements for gcc to at least 4.8.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-27 10:00:29 +01:00
Yann E. MORIN
9c0c7846cd support/dependencies: don't check for python on the host
We no longer have anything that needs it during the build, so we don't
require it anymore.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-26 22:52:15 +02:00
Yann E. MORIN
2496189a42 core: drop check-uniq-files
Back a few years ago, when we were starting to think about top-level
parallel build, we were not sure how to deal with packages that
installed the same files, so we wanted to catch the situation to assess
how prevalent that was, before we decided what to do and how to address
it.

However, the trend nowadays is that packages will install in a
per-package target/ (and staging/ and host/), and the final directories
will be assembled in a reproducible (alphabetical) order, so if two
packages install the same file, the last one will win (as is currently
the case).

Besides, check-uniq-files reports loads of spurious errors when packages
get reinstalled (e.g. during development).

Finally, check-uniq-files is the only script called during the build,
that is written in python.

So, get rid of check-uniq-files.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-26 21:19:07 +02:00
Peter Korsgaard
1cab59a5e8 package/docker-compose: bump version to 1.24.1
Fixes https://gitlab.com/buildroot.org/buildroot/-/jobs/332656041

The recent bump of a number of python packages broke docker-compose, as
docker-compose specifies both minimum and maximum versions for (most of) its
dependencies:

Dependencies of docker-compse 1.20.1 (! = unmet):
cached-property: < 2 (currently 1.51)
docopt: < 0.7 (currently 0.6.2)
! pyyaml: < 4.0, patched to < 4.3 (currently 5.1.2)
requests: < 2.19, patched to < 3 (currently 2.22.0)
! texttable: < 0.10 (currently 1.6.2)
websocket-client: < 1.0 (currently 0.56.0)
! docker: < 4.0 (currently 4.1.0)
dockerpty: < 0.5 (currently 0.4.1)
six: < 2 (currently 1.12.0)
jsonschema: < 3 (currently 2.5.1)
enum34: < 2 (currently 1.1.6)
backports.ssl-match-hostname: >= 3.5 (currently 3.7.0.1)
ipaddress: >= 1.0.16 (currently 1.0.23)

To fix this, bump docker-compose to the most recent release (1.24.1).  This
is unfortunately not enough, as our docker, pyyaml, requests and texttable
packages are too new, so add 3 patches from upstream to relax the version
checks of dependencies.  Notice that patch 0003 is from
https://github.com/docker/compose/pull/6623 and has not been merged yet.

Discussions around the problem of these maximum versions of the dependencies
and the fact that all downstream users have to patch it is ongoing here:

https://github.com/docker/compose/issues/6756

docker-compose 1.24.1 added a requirement for ssh support in python-docker in:
7b82b2e8c7

So add a dependency for python-paramiko and update the toolchain dependency
for C++ (from python-paramiko -> python-cryptography) and adjust the
toolchain configuration of the runtime test to match.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 20:17:14 +02:00
Francois Perrad
76b4fcb5dc support/testing: add luvi test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:33:12 +02:00
Francois Perrad
d1e01107a7 support/testing: add lua-sdl2 test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:31:36 +02:00
Francois Perrad
308ef3166b support/testing: add lua-gd test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:31:03 +02:00
Francois Perrad
eaa2c3a6f7 support/testing: add lzlib test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:30:58 +02:00
Ricardo Martincoski
f3c4a9e1cd support/testing: test_syslog_ng: improve commands
There is no need for double grep, so choose a better regexp. Use &&
instead of ; between commands so the sequence of commands fail faster.

Break the last sequence of commands in 2 calls run() so the proper
return code can be tested for each.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-16 21:33:04 +02:00
Ricardo Martincoski
d3be2087d0 support/testing: add tmux test
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-12 15:37:51 +02:00
Francois Perrad
602f0061ff support/scripts/graph-depends: cut on host-ccache
When selected, host-ccache is a dependency of almost all packages.
As such, it clutters the dependency graph uselessly.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-02 21:07:14 +02:00
Marcin Niestroj
f8b8a9c529 support/testing: add turbolua test
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-02 20:57:08 +02:00
Francois Perrad
101ae29c07 support/testing: add lua-cqueues test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-29 09:15:08 +02:00
Carlos Santos
9e546440d3 support/scripts/check-kernel-headers.sh: use a trap to remove the temporary file
The POSIX specification defines a 'trap <action> EXIT' mechanism that is
useful to perform clean-up actions in shell scripts. A trap has two main
advantages over hand-crafted clean-up mechanisms:

- It runs even if the process is terminated by a SIGTERM.
- It runs even if the script stops due to a pipeline failure (set -e).

Now we can make the script to stop immediately if a compilation error
occurs, instead of letting it try to run an unexisting program.

This change may appear to be overkill but Buildroot is an open source
project and each piece of code is a potential learning tool for other
developments. We must strive to provide good examples.

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Acked-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:07:29 +02:00
Carlos Santos
6136765b23 toolchain: generate check-headers program under $(BUILD_DIR)
Some installations mount /tmp with the 'noexec' option, which prevents
running the program generated there to check the kernel headers.

Avoid the problem by generating the program under $(BUILD_DIR), passed
as the first argument to check-kernel-headers.sh.

We could globally export a TMPDIR environment variable with some path
under $(BUILD_DIR) but such solution would be too intrusive, depriving
the user from the freedom to set TMPDIR at his will (or needs).

Fixes: https://bugs.busybox.net/show_bug.cgi?id=12241

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:07:24 +02:00
Thomas Petazzoni
f89871e810 support/config-fragments: fix br-riscv{32,64} toolchain fragments
Since commit aee39cbf27 ("arch/riscv:
set the default float ABI based on ISA extensions"), RISC-V 32/64 use
the lp32d/lp64d ABIs by default. But our pre-built external toolchains
were built with the LP32/LP64 ABI.

Building with lp32d/lp64d gcc flags, but a toolchain built with the
LP32/LP64 ABI causes a number of failures such as:

  /home/mark/buildroot-test/instance-1/output/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/gnu/stubs.h:11:11: fatal error: gnu/stubs-lp64d.h: No such file or directory

or:

  /home/mark/buildroot-test/instance-1/output/host/opt/ext-toolchain/bin/../lib/gcc/riscv64-buildroot-linux-gnu/7.4.0/../../../../riscv64-buildroot-linux-gnu/bin/ld: /tmp/cc2BTtFE.o: can't link hard-float modules with soft-float modules
  /home/mark/buildroot-test/instance-1/output/host/opt/ext-toolchain/bin/../lib/gcc/riscv64-buildroot-linux-gnu/7.4.0/../../../../riscv64-buildroot-linux-gnu/bin/ld: failed to merge target specific data of file /tmp/cc2BTtFE.o

So let's fix our config fragments to reflect the ABIs those toolchains
were built with.

Fixes:

  http://autobuild.buildroot.net/results/a3959b0613cf561059483abc580b144be4817d1a/ (libsepol)
  http://autobuild.buildroot.net/results/3db50d8a0a913413b2198d6c301419136d2d22a7/ (attr)
  http://autobuild.buildroot.net/results/7780fada05b8440ae3e97618615624a6a2dac03f/ (libusb)
  and many others

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-23 21:56:24 +02:00
Sergey Matyukevich
2a220de466 binaries-marvell: bump version and switch to common firmware
Since release 18.12 binaries-marvell repository provides
common firmware supporting both A7K and A8K SoC families.

This commit bumps package version to 18.12 and removes
platform specific binary selections from Config.in.
Single firmware image suitable for both A7K and A8K
platforms is now specified in mk file explicitely.

Legacy handling is not needed, as configs which did have
the option set will continue to work without change.

Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
[Arnout:
 - Remove BINARIES_MARVELL_IMAGE entirely;
 - Add remark about legacy handling;
 - Remove the deprecated option from the defconfigs and test that use
   it.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-09-21 18:31:26 +02:00
Adrian Perez de Castro
e0c879509d support/dependencies/dependencies.sh: check for JSON:PP Perl module
The JSON::PP Perl module is used at build time by the webkitgtk and
wpewebkit packages.

Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-17 22:36:42 +02:00
Thomas Petazzoni
ffcd34af07 support/scripts/pkg-stats: simplify Git commit id retrieval
As suggested by Baruch Siach, using "git rev-parse HEAD" is a lot
simpler than playing around with "git log" to just retrieve the commit
id corresponding to the current HEAD.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-15 16:10:22 +02:00
Thomas Petazzoni
3f08ffa423 support/scripts/pkg-stats: extract current commit id, not master
pkg-stats extracts the Buildroot commit id from which the package
information was collected. However, when doing so, it always assumes
we're using the master branch, by running "git log master".

But in fact, pkg-stats can be run from any branch/tag, so it makes a
lot more sense to use "git log HEAD".

Cc: victor.huesca@bootlin.com
Cc: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-12 18:53:14 +02:00
Peter Korsgaard
d8e6851f11 Merge branch 'next' 2019-09-03 15:03:02 +02:00
Peter Korsgaard
1fcdfbfb8a Update for 2019.08
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-09-01 23:06:01 +02:00
Yann E. MORIN
20cbf17e0a support/graph-size: reorder colours assigned to sizes
Now that we can order packages from biggest to smallest, it makes sense
to assign the most aggressive colours to the biggest packages.

As such, reorder the current colours so that we have, in order:
  - red-ish
  - orange-ish
  - yellow-ish
  - purple-ish
  - eggplant-ish (is that even a colour? :-] )
  - some-indeterminate-blue-ish
  - dark-green-ish
  - light-green-ish

For the previous, smallest-first ordering, it does not matter much what
the ordering is: the actual colours are still somewhat-unpredictably
assigned to packages, depending on the cut-off limit...

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:51:47 +02:00
Yann E. MORIN
33c1ef88f8 support/graph-size: add option to sort packages in reverse size order
Currently, the packages are sorted smallest first, and biggest last
(with unknown and others second-to-last and last, resp.).

Add an option to invert the ordering (but keeping unknown and others at
their current positions).

This has the nice side effect that we can now control the colours
assigned to the biggest package(s), as the colours are cycled from the
first to the last. Currently, the biggest packages gets a redish colour,
which is appropriate, but the second gets a greenish one, which is not
as appropriate (but changing that can come later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:50:05 +02:00
Yann E. MORIN
1dbce133db support/graph-size: add option to report size with IEC prefixes
When dealing with embedded devices, storage is more often than not some
kind of flash device, on which the memory is usually counted as powers
of 1024 instead of powers of 1000. As such, people may prefer reports
using IEC prefixes [0] instead of the SI prefixes.

Add an option to that effect.

We use argparse's ability to use custom actions [1] [2], to provide a
set of options that act on a boolean, but has a single help entry and
internally ensures consistency of the settings. We could have been using
the more conventional store_true/store_false actions instead, but that
would have meant either two help entries, one for each set of options,
and/or some logic after parse_args() to check the validity of the
settings.

[0] https://en.wikipedia.org/wiki/Binary_prefix
[1] https://docs.python.org/2/library/argparse.html#action
[2] https://docs.python.org/2/library/argparse.html#argparse.Action

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:49:22 +02:00
Yann E. MORIN
e9cdabee71 support/graph-size: add option to change percentage to group in Others
Currently, we group packages that contribute less then 1%, into the
"Other" category.

However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.

Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.

Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
 - remove empty base class definition from Config;
 - use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:44:27 +02:00
Yann E. MORIN
3fc3c4ac99 support/graph-size: display human-readable size
Currently, we forcibly report sizes in multiple of Kilobytes. In some
big configurations, the sizes of the system as a whole, as well as that
of individual packages, may exceed megabytes, and when some artistic
assets get used, even the gigabyte may get exceed.

These big sizes are not easy to read when expressed in kilobytes.

Additionally, some very small packages might have sizes below the
kilobyte (and when we can specify the cut-off grouping size, they may
get reported), and thus the size displayed for those would be 0 kB.

Add a helper function that can format a floating-point size into a
string with all the appropriate formatting:

  - there are at least 3 meaningfull digits visible, i.e. we display
    "3.14" or "10.4" instead of just "3" or "10", but for big number we
    don't care about too many precision either, so we report "100" or
    "1000", not "100.42" or "1000.27";

  - the proper SI prefix is appended, if needed.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:15:35 +02:00
Yann E. MORIN
e8de561436 support/graph-size: report 'Unknown" after all packages, but before "Others"
Currently, the "unknown" category may be reported anywhere, so it does
not really stand out when there are a lot of packages in the graph.

Move it towards the end, but right before the "other" category, so that
it is a bit more visible. Like for Others, don't report it if its size
is zero.

Also, make it title case (i.e. "Unknown" instead of "unknown").

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:12:37 +02:00
Yann E. MORIN
c68ee73924 support/graph-size: don't report "Others" if size is zero
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:08:18 +02:00
Yann E. MORIN
a2d20ca613 support/graph-size: introduce main()
It is nicer overall to have a main() function, like all our other
scripts tend to have too.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:08:01 +02:00
Yann E. MORIN
cecaf7001f support/graph-size: fix flake8 warnings
There are three E501 warnings returned by flake8, when run locally,
because we enforce a local 80-char limit, but that are not reported by
the gitlab-ci jobs because only a 132-char limit is required there.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:07:50 +02:00
Victor Huesca
152776b420 support/testing: new test for root password
Add support to test that the root passowrd is working as expected.
- Buildtime test: Check the hash present in the generated '/etc/shadow'.
- Runtime test: Build an armv7 image and try to login with a password.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-13 22:30:41 +02:00
Ricardo Martincoski
620c98a214 support/testing: factor out run_cmd_on_host
Currently many test cases call subprocess.check_output on their own.
Factor out that code to an infra method so the call get standardized.

This will be handful when switching the test infra to use Python 3.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:58 +02:00
Ricardo Martincoski
1b2928201f support/testing: do not print non-utf-8 chars on cbor test
The python-cbor sample script to be run on the target dumps a binary
variable to the target stdout. This is done for debug purposes-only, in
the case the test on test infra fails.
This non-utf-8 is currently silently ignored by the Emulator class /
pexpect.spawn from the test infra because the infra uses Python 2, that
in turn do not differentiate between string and byte data.

Make the code Python3-friendly (Python 3 in the host) by doing the right
thing and encoding the data before printing it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:55 +02:00
Ricardo Martincoski
8de4291bd2 support/testing: check ubi image before booting
Do the same as other fs tests and minimally check the ubi image before
booting.
The call to 'file' was already there, but the output wasn't tested for
some unknown reason. Add the assert for the output of the command.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:53 +02:00
Ricardo Martincoski
a565917046 support/testing: provide entropy to perl tests
Newer versions of perl-io-socket-ssl require entropy.
Switch to use armv5 builtin kernel that already provides entropy for all
perl tests.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139402

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Francois Perrad <francois.perrad@gadz.org>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski
eb6b0fd87a support/testing: provide entropy to lua tests
Newer versions of lua-http require entropy.
Switch to use armv5 builtin kernel that already provides entropy for all
lua tests.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139374
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139376

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Francois Perrad <francois.perrad@gadz.org>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski
d23056ffe1 support/testing: make crossbar to use umsgpack
Since version 19.3.3, python-autobahn assumes that when CPython is used
msgpack will be used as well [1]. But it still allows the user to
override this behavior by setting an environment variable [2] to use
umsgpack.

Make the test to explicitly use umsgpack since it is part of the minimal
config (python-crossbar selects python-u-msgpack).

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139430

[1] ea019b8042
[2] fe70ceebe0

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski
ef5e87bb92 support/testing: make treq tests to ignore warning
Newer versions of python-treq display a warning at runtime when
service_identity is not installed:
"Without the service_identity module, Twisted can perform only
rudimentary TLS client hostname verification."

This warning message confuses the test that looks for another string in
stdout. Make the test ignore other messages while still expecting
"Connection refused".

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139449
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139450

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski
5c39c90ae1 support/testing: fix hardening tests
Since "2467822c85 package/checksec: bump to version 2.1.0" the hardening
tests fail because upstream slightly changed the way the script is
called.
According to README.md: "- All options now require `--$option=$value`
instead of `--$option $value`"

Instead of just replacing '--output json' with '--output=json' take into
account that upstream also changed the usage example to show --format
instead of --output. Both options do exactly the same, but following the
usage example seems to be more future-proof.

Upstream also improved the json output. Now when a file is passed as
parameter, the json has the file name as the main key, instead of the
string "file". Adjust the test cases accordingly.

Fixes:
tests.core.test_hardening.TestFortifyConserv
tests.core.test_hardening.TestFortifyNone
tests.core.test_hardening.TestRelro
tests.core.test_hardening.TestRelroPartial
tests.core.test_hardening.TestSspNone
tests.core.test_hardening.TestSspStrong

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 22:15:18 +02:00
Thomas Petazzoni
dc938c221d support/config-fragments/autobuild: set gcc version for RISC-V toolchains
Prior to b3ba26150d
("toolchain/toolchain-external/toolchain-external-custom: be more
flexible on gcc version"), the default gcc version selected by
Buildroot for custom external toolchain was affected by the
BR2_ARCH_NEEDS_GCC_AT_LEAST_xyz definitions.

Since BR2_riscv selects BR2_ARCH_NEEDS_GCC_AT_LEAST_7, gcc 7.x was the
default gcc version assumed to be used in a custom RISC-V external
toolchain, so our config snippets for RISC-V toolchains were correct.

With b3ba26150d applied, the default gcc
version assumed for custom external toolchains is the latest one
(currently gcc 9.x), while our RISC-V toolchains use gcc 7.x. So we
now need to explicitly give the gcc version used by our RISC-V
toolchains, otherwise the build fails with:

  Incorrect selection of gcc version: expected 9.x, got 7.4.0

Fixes:

  http://autobuild.buildroot.net/results/b872befe1adec2633b9cbcc49bc0eb7619f606c2/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 08:55:10 +02:00
Yann E. MORIN
cfb929fbfa core: allow br2-external trees to provide opensl
Similar to toolchains and jpeg, we now offer a way for br2-external
trees to provide their openssl implementation, which gets included in
the openssl choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN
3b67e8e664 core: allow br2-external trees to provide libjpeg
Similar to toolchains, we now offer a way for br2-external trees to
provide their libjpeg implementation, which gets included in the jpeg
choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN
fa037acee0 core: allow br2-external trees to provide pre-configured toolchains
Since we have a choice for the pre-configured pre-built toolchains,
there is no possbility for a br2-external to provide its own. The
only solution so far for defconfigs in br2-external trees is to use
BR2_TOOLCHAIN_EXTERNAL_CUSTOM and define all the bits by itself...

This is not so convemient, so offer a way for br2-external trees to
provide such pre-configured toolchains.

To allow for this, we now scan each br2-external tree and look for a
specific file, provides.toolchains.in. We generate a kconfig file that
sources each such file, and that generated file is sourced from within
the toolchain choice, thus making the toolchains from a br2-external
tree possible and available in the same location as the ones known to
Buildroot:

    Toolchain  --->
        Toolchain type (External toolchain)  --->
        Toolchain  --->
            (X) Arm ARM 2019.03
            ( ) Linaro ARM 2018.05
            ( ) Custom toolchain
                *** Toolchains from my-br2-ext-tree: ***
            ( ) My custom ARM toolchain
                *** Toolchains from another-br2-ext-tree: ***
            ( ) Another custom ARM toolchain
            ( ) A third custom ARM toolchain

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN
edf32b021c core: split generated kconfig file
Currently, the kconfig part contains two things: the kconfig option
with the paths to br2-external trees, and the kconfig menus for the
br2-external trees.

When we want to include more kconfig files from the br2-external tree
(e.g. to get definitions for pre-built toolchains), we will need to
have the paths defined earlier, so they can be used from the br2-external
tree to include files earlier than the existing menus.

Split the generated kconfig file in two: one to define the paths, which
gets included early in our main Config.in, and one to actually define
the existing menus, which still gets included at the same place they
currently are.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN
0797dae894 core: prepare for generating multiple kconfig fragments
We currently redirect the output of each helper function. This was nice
as long as we were generating single .mk and .in fragments.

But we are soon to need more .in fragments.

So, do the redirection inside the .in helpers.

We do not (currently) need to generate more than one .mk fragment, but
for consistency, do the redirection in the .mk helper too.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
d027cd75d0 core: generate all br2-external files in one go
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.

But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.

And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.

Make it so.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
2130903347 support/scripts/br2-external: drop help for internal helper script
We do not usually provide help for our internal scripts. Besides, such
help has a tendency to bitrot pretty quickly anyway.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:58:46 +02:00
Yann E. MORIN
3617f1350a support/scripts/br2-external: declare missing local variables
Commit b14b02698 (core/br2-external: restore compatibility with old
distros) switched to using 'eval' to emulate associative arrays, for
those distros too old to have bash-4+.

In so doing, it forgot to declare the new local variables in the
respective helper functions.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:58:17 +02:00
Frank Vanbever
07f31ee263 support/cmake: Explicitly set CMAKE_SYSTEM
Some packages test for CMAKE_SYSTEM explicitly[1]

CMAKE_SYSTEM is comprised of CMAKE_SYSTEM_NAME and CMAKE_SYSTEM_VERSION.
It defaults to CMAKE_SYSTEM_NAME if CMAKE_SYSTEM_VERSION is not set[2]

At the point CMAKE_SYSTEM_NAME is set to "Linux" CMAKE_SYSTEM is already
constructed. Setting it explicitly ensures that it is the correct value.

This is because we do set CMAKE_SYSTEM_NAME twice, in fact:

  - first in toolchainfile.cmake, so that we tell cmake to use the
    "Buildroot" platform,

  - second, in the Buildroot.cmake platform definition itself, so that
    we eventually behave like the Linux platform.

We also set CMAKE_SYSTEM_VERSION to 1, and so the real CMAKE_SYSTEM
value should be set to Linux-1 if we were to follow the documentation to
the letter.

However, for Linux, the version does not matter, and in some situations
may even be harmful (that was reported in one of the commits that
introduce Buildroot.cmake and toolchainfile.cmake).

[1] Fluidsynth 0cd44d00e1/CMakeLists.txt (L80)
[2] https://cmake.org/cmake/help/git-master/variable/CMAKE_SYSTEM.html#variable:CMAKE_SYSTEM

Signed-off-by: Frank Vanbever <frank.vanbever@mind.be>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Peter: update commit message with description from Yann]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:17:32 +02:00
Ricardo Martincoski
1ad5e0a6a6 support/testing: test check-package ignores external.mk
Check external.mk is ignored only when in the root path of a
br2-external.

Add a file called external.mk as a fixture to be used by the test case.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
4323fbaf8d support/testing: add test for check-package
Check the basic usage for check-package.

It can be called using either absolute path, relative path or from PATH.
Files to be checked can be passed with either absolute path or relative
path (also including files in the current directory).

Also check it ignores some special files when checking intree files,
i.e. package/pkg-generic.mk, while still generating warnings for out-of-tree
files when called with -b.
In order to allow the later, add an empty line to the Config.in in the
br2-external being tested so the script does generate a warning.

Catches bug #11271.

More tests can be added later, for example compatibility to Python 3.

Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns where appropriate; merge into a single
class.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
8669243da3 support/testing: allow run-tests to be called from anywhere
Currently run-tests must be called from the Buildroot top directory.

Derive the top directory from the script path, so run-tests can be called from
any path.
As a consequence the test infra will always test the repo it belongs to.

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
9609a7c2c2 support/testing: test atop with builtin kernel
Use the new builtin armv5 4.19 kernel to test atop.
The atop package cannot be tested using BASIC_TOOLCHAIN_CONFIG because
it needs kernel headers >= 3.14. So use an updated version of it,
copying the config fragment from
support/config-fragments/autobuild/br-arm-full.config

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski
650e9903a9 support/testing: drop haveged for python tests
The new builtin armv5 kernel 4.19 uses VirtIORNG to provide entropy, so we
no longer need haveged.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508536
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508537
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508542
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508543

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Peter: reword]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski
56e38a705e support/testing: test syslog-ng with builtin kernel
Use the new builtin kernel 4.19 with VirtIORNG to provide entropy to
test syslog-ng.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski
7acb32dabb support/testing: add builtin armv5 kernel 4.19 with entropy
More and more packages being tested by the test infra, e.g. syslog-ng,
need entropy at startup, usually reading from /dev/random.

Some test cases can also depend on a kernel version newer than the
builtin ones already provided by the test infra:
 - 3.11.0 for armv5;
 - 4.0.0 for armv7.

Add a new builtin kernel to be used by such test cases.
Add it for armv5 so most test cases that switch to use this kernel can
keep using BASIC_TOOLCHAIN_CONFIG.
Use the same kernel version and kernel config as qemu_arm_versatile plus
HW_RANDOM_VIRTIO for VirtIORNG to be usable.
Copy the actual binary file from the syslog-ng runtime test at current
master @ 29e1cb8884.

Since there is already a 'kernel-versatile' file on autobuild.buildroot.net
and we must keep it with this name for reproducibility purposes, create a
simple naming convention for newer builtin kernel images and dtb files:
kernel-<defconfig>-<kernel_series_version>
<dtb_name>-<kernel_series_version>.dtb
Pass '-device virtio-rng-pci' to qemu when this kernel is used.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Peter: use this new kernel instead of the old builtin/armv5 kernel]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Thomas Petazzoni
228b0b1f00 support/config-fragments/autobuild: update CSV after musl toolchain rename
In commit a589325405
("support/config-fragments/autobuild: rename br-riscv64-musl config"),
the RISC-V 64-bit musl toolchain config snippet was renamed, but the
toolchain.csv file was not updated accordingly.

Due to this, utils/genrandconfig was no longer able to generate any
configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-03 12:57:46 +02:00
Thomas Petazzoni
b96c343b2d support/config-fragments/autobuild: make br-arm-internal-glibc bleeding edge
We do not have any configuration that tests the very bleeding edge gcc
and binutils versions, so let's change br-arm-internal-glibc to use
the latest version of gcc (9.x right now) and binutils (2.32 right
now). The idea is that this defconfig should be updated to the latest
version of gcc and binutils when their version is bumped.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:33:00 +02:00
Thomas Petazzoni
84f356e375 support/config-fragments/autobuild: update toolchains to 2019.05.1
All toolchains have been rebuilt with Buildroot 2019.05.1. A number of
toolchains are now using Linux headers 5.1 instead of 4.19, because
5.1 is now the default version.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:32:40 +02:00
Thomas Petazzoni
a589325405 support/config-fragments/autobuild: rename br-riscv64-musl config
For internal toolchains, we have a policy of naming the files with
"internal", to clearly distinguish them from external toolchain
configurations.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:32:19 +02:00
Francois Perrad
1a49dcb528 support/testing: improve test_luasyslog
Lua has a builtin lsyslog module, so let's test this one as well.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-02 17:35:44 +02:00
Mark Corbin
1003af2ebe support/config-fragments: add RISC-V 64-bit musl autobuild config
Add a RISC-V 64-bit autobuild configuration for the internal
toolchain with musl.

Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-01 23:55:40 +02:00
Ricardo Martincoski
450863097f support/testing: use virtio-rng-pci to test syslog-ng
Recent versions of syslog-ng need some entropy on startup.
So use VirtIORNG to provide it. In order to accomplish this:
 - build the kernel containing the driver;
 - pass '-device virtio-rng-pci' to qemu.

Use the same kernel version and kernel config as qemu_arm_versatile.
It already has PCI enabled but it does not have HW_RANDOM_VIRTIO, so add
a defconfig fragment to enable the drivers.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/259856394

At the same time, fix a typo (missing '#') that resulted in the
generation of root.tar. This file is not used in the test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-01 20:31:54 +02:00
Francois Perrad
0b32cd888e package/luaposix: bump to version 34.1.1
the module std.normalize is no longer a dependency

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:11:33 +02:00
Victor Huesca
294fc3218c support/scripts/pkg-stats: retrieve packages latest version using processes
The major bottleneck in pkg-stats is the time spent waiting for
answers from remote servers. Two functions involve such communication
with remote servers:

- 'check_package_urls' which checks that each package upstream website
  is up, it is efficient due to the use of process-pools thanks to
  Matt Weber.

- 'check_package_latest_version' which fetches the latest package
  version from release-monitoring, it uses a http-pool but runs
  sequentially.

This patch extends the use of process-pools to 'check_latest_version'.
Due to some limitations of multiprocess callbacks, this patch loses
the overall progress of packages in favour of just the current package
name.

Runtimes for this function are ~3m vs ~25m for the linear version.
Tested on an i7 7500U (2/4 cores/threads @3.5GHz) with 15ms ping.

Note: There have already been work trying to parallelize this function
using threads but there were a failure on some configurations [1].
This implementation rely on a dedicated module already in use on this
script, so it's unlikely to see failure with this version.

[1] http://lists.busybox.net/pipermail/buildroot/2018-March/215368.html

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:04:09 +02:00
Victor Huesca
3938afe1b5 support/scripts/pkg-stats: fix flake8 issues
Fixes:
 - blank space before ':'
 - unused 'o' variable left from a previous patch
 - bad continuous alignment

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:04:09 +02:00
Victor Huesca
46190a36d9 support/scripts/pkg-stats: improve 'package_init_make_info'
The pkg-stats calls 3 times `make` to get a bunch of variables. These
variables can be obtained in only one make invocation.  This patch
replaces the three calls by just one and adjusts the parsing logic
accordingly.

Note: another option suggested by Arnout would be to run `make
show-info` that produces a json with the necessary variables.  This
would avoid the duplicated effort done in pkg-stats and pkg-utils and
allow to add other infos to pkg-stats like dependencies, reversed
dependencies or if the package is virtual.

In order to use this method, the following changes are required in
pkg-generic's show-info:

 - include license_files;
 - have an option to run it on *all* packages, not just the selected
   ones.

This patch take the simplest approach of only factorizing the make
calls as it requires less changes.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 14:27:23 +02:00
Thomas Petazzoni
2a16a0ff08 support/scripts/pkg-stats: rename dump_gen_info()
Since it's used only for the HTML output, and all other functions used
for HTML output are prefixed by dump_html, let's do so for
dump_gen_info() as well by renaming it to dump_html_gen_info().

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Victor Huesca
3c9d408207 support/scripts/pkg-stats: factorize date and commit
The 'dump_html' and 'dump_json' both include commit infos as well as the
current date. It make more sense to retrieve these information once.
This patch simply does this factorization.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Victor Huesca
500e1d6241 support/scripts/pkg-stats: add support for json output
Pkg-stats is a great script that get a lot of interesting info from
buildroot packages. Unfortunately it is currently designed to output a
static HTML page only. While this is great to include on the
buildroot's website, the HTML is not designed to be easily parsable and
thus it is difficult to reuse it in other scripts.

This patch provide a new option to output a JSON file in addition to the
HTML one.

The old 'output' option has been renamed to 'html' to distinguish from
the new 'json' option.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00