Commit Graph

1215 Commits

Author SHA1 Message Date
Thomas Petazzoni
09a71e6a75 support/scripts/cpedb.py: drop CPE XML database caching
Currently, the CPE XML database is parsed into a Python dict, which is
then pickled into a local file, to speed up the processing of further
invocations.

However, it turns out that since the initial implementation, we have
switched the XML parsing from the out of tree xmltodict module to the
standard ElementTree one, which has made the parsing much faster. The
pickle caching only saves 6 seconds, on something that takes more than
13 minutes total.

In addition, this pickle caching consumes a significant amount of RAM,
causing the Python process to be OOM-killed on a server with 4 GB of
RAM.

So let's just drop this caching entirely.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-14 10:05:00 +01:00
Sergey Matyukevich
653b127d1e support/testing: TestATFAllwinner needs python3 for u-boot
New U-Boot version needs Python 3.x for pylibfdt.

Fixes:
- https://gitlab.com/buildroot.org/buildroot/-/jobs/1019385909

Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-13 22:46:27 +01:00
Thomas Petazzoni
e1c0360606 support/config-fragments/autobuild: use Bootlin toolchain for RISC-V 64 musl
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 64/musl, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-05 09:27:37 +01:00
Thomas Petazzoni
63485a6c1e support/config-fragments/autobuild: use Bootlin toolchain for RISC-V 64 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 64/glibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-05 09:25:23 +01:00
Thomas Petazzoni
032cf04fe7 support/config-fragments/autobuild: update Buildroot toolchains to 2020.11.2
Most of the toolchains now use gcc 9.x and kernel headers 5.9, instead
of gcc 8.x and kernel headers 5.4.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:47:02 +01:00
Thomas Petazzoni
dcb2747d65 support/config-fragments/autobuild: use Bootlin toolchain for Xtensa uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test Xtensa/uclibc, use a pre-built Bootlin toolchain.

To be noted: that fragment was in fact already using a Bootlin
bleeding-edge toolchain, because BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y is
missing from the fragment:

    $ cat support/config-fragments/autobuild/br-xtensa-full.config >.config

    $ make olddefconfig

    $ grep BOOTLIN .config
    BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
    BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS=y
    BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_XTENSA_LX60_UCLIBC_BLEEDING_EDGE=y
    # BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_XTENSA_LX60_UCLIBC_STABLE is not set

The original fragment was supposed to use a stable toolchain, so we
switch to explictly use a stable Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - add blurb about missing BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:42:51 +01:00
Thomas Petazzoni
6bed28e996 support/config-fragments/autobuild: use Bootlin toolchain for x86-64 musl
Instead of using an external toolchain built specifically for the
autobuilders to test x86-64/musl, use a pre-built Bootlin toolchain.

The previous configuration was for an Atom platform, but the Bootlin
toolchains only provide a Core i7 configuration. Since this is close
enough, we change to use this Core i7 configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:42:39 +01:00
Thomas Petazzoni
5679e58c9f support/config-fragments/autobuild: use Bootlin toolchain for x86-64 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test x86-64/uclibc, use a pre-built Bootlin toolchain.

The previous configuration was for Core2 platform, but the Bootlin
toolchains only provide a Core i7 configuration. Since this is close
enough, we change to use this Core i7 configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:42:39 +01:00
Thomas Petazzoni
9e56f77fdf support/config-fragments/autobuild: use Bootlin toolchain for SPARC64 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test SPARC64/glibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:42:39 +01:00
Thomas Petazzoni
1348c569d0 support/config-fragments/autobuild: use Bootlin toolchain for SPARC uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test SPARC/uclibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:42:39 +01:00
Thomas Petazzoni
e022d47908 support/config-fragments/autobuild: use Bootlin toolchain for SH4 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test SH4/uclibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: drop BR2_sh4=y which is the default]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:41:52 +01:00
Thomas Petazzoni
fc266b1c1a support/config-fragments/autobuild: use Bootlin toolchain for RISC-V 32 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test RISC-V 32/glibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:09 +01:00
Thomas Petazzoni
563285a63a support/config-fragments/autobuild: use Bootlin toolchain for PowerPC e500mc uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test PowerPC e500mc/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:08 +01:00
Thomas Petazzoni
15af077d8e support/config-fragments/autobuild: use Bootlin toolchain for PowerPC64le Power8 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test PowerPC64le Power8/glibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:07 +01:00
Thomas Petazzoni
739cb9c9d5 support/config-fragments/autobuild: use Bootlin toolchain for OpenRISC uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test OpenRISC/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:06 +01:00
Thomas Petazzoni
b18e923174 support/config-fragments/autobuild: use Bootlin toolchain for NIOS2 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test nios2/glibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:05 +01:00
Thomas Petazzoni
625cbe23b3 support/config-fragments/autobuild: use Bootlin toolchain for mipsel uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test mipsel/uclibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:03 +01:00
Thomas Petazzoni
577ddc907f support/config-fragments/autobuild: use Bootlin toolchain for mipsel32r6 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test mipsel32r6/glibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:02 +01:00
Thomas Petazzoni
899e374730 support/config-fragments/autobuild: use Bootlin toolchain for Microblaze EL uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test Microblaze EL/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:01 +01:00
Thomas Petazzoni
5e19a011fd support/config-fragments/autobuild: use Bootlin toolchain for m68k 5208 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test m68k 5208/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:37:00 +01:00
Thomas Petazzoni
f0ab9611a0 support/config-fragments/autobuild: use Bootlin toolchain for m68k 68040 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test m68k 68040/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:59 +01:00
Thomas Petazzoni
8659946e1c support/config-fragments/autobuild: use Bootlin toolchain for ARMv7-M uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv7-M/uclibc, use a pre-built Bootlin
toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:57 +01:00
Thomas Petazzoni
75569c9fa9 support/config-fragments/autobuild: use Bootlin toolchain for ARMv7 musl
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv7/musl, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:56 +01:00
Thomas Petazzoni
673e39a043 support/config-fragments/autobuild: use Bootlin toolchain for ARMv7 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test ARM Cortex-A9/glibc, use a pre-built Bootlin
toolchain. Since this was meant to test very recent version of
toolchain components, we use the bleeding edge toolchain variant.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:55 +01:00
Thomas Petazzoni
ff203421db support/config-fragments/autobuild: use Bootlin toolchain for ARMv5 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test ARMv5/uclibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:54 +01:00
Thomas Petazzoni
fbba59f478 support/config-fragments/autobuild: use Bootlin toolchain for ARCle HS38 uclibc
Instead of using an external toolchain built specifically for the
autobuilders to test ARCle HS38/uclibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:53 +01:00
Thomas Petazzoni
133e57cc2c support/config-fragments/autobuild: use Bootlin toolchain for AArch64 glibc
Instead of using an external toolchain built specifically for the
autobuilders to test AArch64/glibc, use a pre-built Bootlin toolchain.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-03 22:36:52 +01:00
Thomas Petazzoni
201e74bf9d support/scripts/pkg-stats: check CPE existence in CPE dictionnary
This commit extends pkg-stats to leverage the recently introduced
CPEDB class to verify that the CPEs provided by Buildroot packages are
indeed known in the official CPE dictionnary provided by NVD.

Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-02-02 20:16:59 +01:00
Matt Weber
ba8e4767d0 support/scripts/cpedb.py: new CPE XML helper
Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.

 - Defines the CPE as a object with operations / formats
 - Processing of CPE dictionary

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-02-02 20:16:49 +01:00
Peter Korsgaard
97f1d5d151 support/scripts/pkg-stats: get_config_packages(): use dict.values()
There is no need to get both the key and the value out of the dict if the
key is not used, so use dict.values() instead.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-02-02 16:34:10 +01:00
Fabrice Fontaine
f7922a57f5 Replace LIBFOO_CPE_ID_VERSION_MINOR by LIBFOO_CPE_ID_UPDATE
Replace LIBFOO_CPE_ID_VERSION_MINOR by LIBFOO_CPE_ID_UPDATE to better
"comply" with the official "Well-Formed CPE Name Data Model" parameters:
 - https://csrc.nist.gov/publications/detail/nistir/7695/final
 - https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 23:21:38 +01:00
Thomas Petazzoni
c9a1a1eb91 support/scripts/pkg-stats: properly handle host packages with -c option
In commit 7a607dab33
("support/scripts/pkg-stats: support generating stats based on
configured packages"), we added a -c option to pkg-stats to generate a
report based on the list of packages enabled in the configuration,
rather than for all packages.

This is done based on the list of packages returned in JSON format by
"make show-info". However, we use the keys of the JSON dict returned
by "make show-info", which include the host- prefix of host
packages. Due to this, none of the host packages are currently
matching and therefore they are not reported in the pkg-stats -c
output.

This commit fixes that by using the recently introduced "name"
property in the "make show-info" JSON dict.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use anonymous '_' for unused variable]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-01-31 23:01:21 +01:00
Peter Korsgaard
a93208ccec Update for 2020.11.2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 551cb63007)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-01-31 22:44:08 +01:00
Thomas Petazzoni
52a53c97e5 support/scripts/pkg-stats: fix flake8 warnings
Fixes:

support/scripts/pkg-stats:148:17: E741 ambiguous variable name 'l'
support/scripts/pkg-stats:379:9: E741 ambiguous variable name 'l'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:12 +01:00
Thomas Petazzoni
67768e97b0 support/scripts/pkg-stats: drop unused --cpeid option
The --cpeid option was mistakenly introduced by commit
92e7089a8c ("support/script/pkg-stats:
show CPE ID in results") but is in fact not necessary.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:11 +01:00
Thomas Petazzoni
cad5a69f83 support/scripts/pkg-stats: drop unused cpeid_name() function
The cpeid_name() function is not used anywhere, drop it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:00 +01:00
Thomas Petazzoni
68364a4b58 support/scripts/pkg-stats: fix the status reporting of CVEs
Since commit bd665d182c
("support/scripts/pkg-stats: improve rendering of CVE information"),
we have better reporting of CVE related information, based on
pkg.status['cve']. However, this commit broke pkg-stats when the
--nvd-path option is not passed, and therefore no CVE information is
available.

This commit fixes that, by making use of the is_status_ok(),
is_status_error() and is_status_na() methods recently introduced.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:02:42 +01:00
Thomas Petazzoni
7f83ad7467 support/scripts/pkg-stats: improvements in is_status_*() methods
Make is_status_ok() work when the given status name is not even listed
in the status dict. This will be necessary for following commits.

Introduced similar methods for the error and na status, which will be
used in following commits.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:02:41 +01:00
Fabrice Fontaine
4b6202f721 Replace LIBFOO_CPE_ID_NAME by LIBFOO_CPE_ID_PRODUCT
Replace LIBFOO_CPE_ID_NAME by LIBFOO_CPE_ID_PRODUCT to better "comply"
with the official "Well-Formed CPE Name Data Model" parameters:
 - https://csrc.nist.gov/publications/detail/nistir/7695/final
 - https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-21 22:43:24 +01:00
Thomas Petazzoni
40bb37bd70 utils/getdeveloperlib.py: use relative paths for files
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.

parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.

This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.

Before this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]

$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": []
}

After this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
  "Floris Bos <bos@je-eigen-domein.nl>",
  "Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": [
    "Christophe Priouzeau <christophe.priouzeau@st.com>"
  ]
}

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:08 +01:00
Thomas Petazzoni
ffb2620405 support/scripts/pkg-stats: fix Python 3.8 deprecation warning
With Python 3.8, the following deprecation warnings are emitted:

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:418: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:536: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

The correct way to pass coroutines is to use asyncio.create_task(),
but this is rather new method (Python 3.7), and using it breaks
compatibility with older Python versions. As suggested at
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task,
use the more cryptic, but also more compatible asyncio.ensure_future()
method.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:05 +01:00
Thomas Petazzoni
824032d168 support/scripts/pkg-stats: import cve module only when needed
The cve module needs ijson, which may not be installed. Since cve
matching is only enabled when --nvd-path is passed, it is a bit silly
to error out about ijson being missing if it's not used.

So instead of unconditionally importing the cve module, only do it
conditionally.

However, instead of doing it right at the point where it is used, we
do it at the beginning of the main() function. Indeed, if the cve
module is needed but cannot be imported, we want to error out
immediately rather than doing a whole bunch of things, and failing on
the user later on in the middle of the pkg-stats execution.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:04 +01:00
Thomas Petazzoni
01533244a5 support/config-fragments/autobuild/bootlin-x86-64-glibc: use toolchain-external-bootlin
Until now the bootlin-x86-64-glibc was using a Bootlin toolchain as a
custom external toolchain. However, now that we have the
toolchain-external-bootlin package explicitly supporting Bootlin
toolchains as known toolchain profiles, it makes sense to use
that. Indeed, this will ensure that this autobuilder configuration
will use the latest available version of the Buildroot toolchain for
x86-64 glibc.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-01-17 10:29:17 +01:00
Sergey Matyukevich
5d54214049 support/testing: fix TestATFAllwinner test
Fix U-Boot config snippet in TestATFAllwinner. Bump U-Boot
version to fix DTC build on hosts with gcc 10 and add
pylibfdt dependency.

Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-16 22:35:20 +01:00
Sergey Matyukevich
22188ff6e0 support/testing: remove TestATFMarvell
There were only two users of Marvell ATF: SolidRun MacchiatoBin board
and ClearFrog GT 8k board. After mv-ddr-marvell package update both
boards switched to upstream ATF. Remove tests for now unused
Marvell ATF.

Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-16 22:35:18 +01:00
Yann E. MORIN
ec50e407be support/dependencies: drop check for maximal tar version
So far, we checked that the tar present on the host was at most tar
1.29, because tar 1.30 changed the way it generates archives.

Having a maximum tar version requirement meant that we would eventually
always have to build our own host-tar, as distributions are updating
the version they use.

But now, we have found a way to generate reproducible archives starting
with tar 1.27 onward, so we no longer need the check for a maximum tar
version, so we can drop that requirement.

Note: this is semantically a revert of b8fa273d50 (check-host-tar.sh:
blacklist tar 1.30+), but keeping the new, mostly-linear code-path.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
2021-01-10 22:06:58 +01:00
Yann E. MORIN
c043ecb20c support/download: change format of archives generated from svn
Like we recently did for git, switch the archives generated from
subversion to be reproducible whatever the tar version.

We have no in-tree users of the svn backend which also has hashes,
so no hash to update.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
2021-01-10 22:06:58 +01:00
Yann E. MORIN
ccfae17015 support/download: cleanup svn backend
Commit 89f5e9893 (support/download/svn: generate reproducible svn
archives) did what it said, but can be siplified a bit.

Indeed, we are doing an svn export, so we won't have any of the .svn
directories, neither at the root of the extract, nor in any of the
sub-directories.

As such, we do not need to filter them out  when we generate the list
of files to include in the archive.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
2021-01-10 22:06:58 +01:00
Yann E. MORIN
5b95a5dc27 support/download: change format of archives generated from git
Switch to using the tarball helper, that can generate reproducible
archives whatever the tar version >= 1.27.

However, those archives are not identical to the previous ones generated
in the (now-broken) gnu format.

To avoid any clashing between old and new archives, and new and old
Buildroot versions, we need to name the new generated archives
differently from the existing ones.

So, we bump the git-specific format-version to -br1.

The %ci date  has been supported by git back to 1.6.0, released August
2008); it is not strictly ISO8601, but is still accepted as a PAX date
header. The strict ISO8601 placeholder, %cI, was only introduced with
2.2.0, release in November 2014, so too recent to be widely available.

As the format and the names of the archives changes, we need to update
all the hash files with the new names and hashes.

Of all the bootloaders that have a git download method, vexpress-firmware
is the only one to have a hash. Others have no hash files, or they have
explicitly set BR_NO_CHECK_HASH_FOR.

For the packages, linux-headers is the special snowflake, as the git
download is only for custom git tree, so it is excluded from the hash
verification with BR_NO_CHECK_HASH_FOR.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>

    ---8<------8<------8<------8<---
    #!/bin/sh
    # Find and download all packages using git as backend.
    # Manually fix hashes for affected packages.

    # Packages that only have a host variant
    HOST_ONLY='imx-mkimage|mxsldr|netsurf-buildsystem|opkg-utils|prelink-cross|qoriq-rcw|vboot-utils'

    # Packages that have a non-git main _SOURCE, and/or which
    # have BR_NO_CHECK_HASH_FOR for the git _SOURCE
    NOT_GIT='aufs|aufs-util|xenomai|linux-headers'

    export BR2_DL_DIR=$(pwd)/temp-dl-dir

    make defconfig
    make $( git grep -l -E 'SITE_METHOD[[:space:]]*:?=[[:space:]]*git\>|_SITE[[:space:]]*:?=[[:space:]]*git:' \
                boot/vexpress-firmware/ package/ \
            |sed -r -e 's,.*/([^/]+)\.mk,\1,' \
            |sed -r -e '/^('"${NOT_GIT}"')$/d;' \
                    -e 's/^('"${HOST_ONLY}"')/host-\1/;' \
                    -e 's/$/-legal-info/;'
          )

    ---8<------8<------8<------8<---
2021-01-10 22:06:58 +01:00
Yann E. MORIN
cbe95b1a45 support/download: add helper to generate a reproducible archive
We currently need to generate reproducible archives in at least two
locations: the git and svn download backends. We also know of some
future potential use (e.g. the other download backends, like cvs, or
in the upcoming download post-processors for vendoring, like cargo
and go).

However, we are currently limited to a narrow range of tar versions
that we support, to create reproducible archives, because the gnu
format we use has changed with tar 1.30.

As a consequence, and as time advances, more and more distros are,
or will eventually start, shipping with tar 1.30 or later, and thus
we need to always build our on host-tar.

Now, thanks to some grunt work by Vincent, we have a set of options
that we can pass tar, to generate reproducible archives back from
tar-1.27 and up through tar-1.32, the latest released version.

However, those options are non-trivial, so we do not want to have
to repeat those (and maintain them) in multiple locations.

Introduce a helper that can generate a reproducible archive from
an input directory.

The --pax-option, to set specific PAX headers, does not accept
RFC2822 timestamps which value are too away from some fixed point
(set atcompile-time?):
    tar: Time stamp is out of allowed range

However, the same timestamps passed as strict compliant ISO 8601 are
accepted, so that's what we expect as a date format.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>

    ---8<------8<------8<------8<---
    # Here is a Makefile used to test all the versions of tar, with
    # different output formats and different sets of options:
    # Versions prior to 1.27 do not build on recent machines, because
    # 'gets()' got removed (rightfully so), so don't count them as
    # candidates.
    VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32
    DATE = Thu 21 May 2020 06:44:11 PM CEST

    TARS = \
    	$(patsubst %,test_gnu_%.tar,$(VERSIONS)) \
    	$(patsubst %,test_posix_%.tar,$(VERSIONS)) \
    	$(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS))

    all: $(TARS)
    	sha1sum $(^)

    .INTERMEDIATE: test_%.tar
    test_gnu_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=gnu \
    		-T list \
    	>$(@)
    test_posix_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=posix \
    		-T list \
    	>$(@)
    test_posix_paxoption_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=posix \
    		--pax-option='delete=atime,delete=ctime,delete=mtime' \
    		--pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \
    		-T list \
    	>$(@)

    list: .FORCE
    list: test
    	(cd test && find . -not -type d ) |LC_ALL=C sort >$(@)

    LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng
    test: .FORCE
    test:
    	rm -rf test
    	mkdir -p test/bar
    	echo foo >test/Foo
    	echo bar >test/bar/Bar
    	ln -s bar/Bar test/buz
    	echo long >test/Very-$(LONG)-filename
    	ln test/Very-$(LONG)-filename \
    	   test/short

    .PRECIOUS: tar.%
    tar.%: tar-%
    	cd $(<) && ./configure
    	$(MAKE) -C $(<)
    	install -m 0755 $(<)/src/tar $(@)

    .PRECIOUS: tar-%
    tar-%: tar-%.tar.gz
    	tar xzf $(<)

    .PRECIOUS: tar-%.tar.gz
    tar-%.tar.gz:
    	wget "https://ftp.gnu.org/gnu/tar/$(@)"

    .FORCE:

    clean:
    	rm -rf tar-* tar.* test_* test list
    ---8<------8<------8<------8<---
2021-01-10 22:04:33 +01:00