Commit Graph

1600 Commits

Author SHA1 Message Date
Yann E. MORIN
a39116b5db support/download: catch post-process errors
Fixes:
http://autobuild.buildroot.net/results/12a/12a63ae177fe3ed0c9a1ef2fa01870f334f36b0f/

Currently, when the post-process helper fails while downloading from
upstream, there is no fallback to the backup mirror.

In case the post-process helper fails, we must consider that to be a
download failure, so we must bail out as if the download backend itself
did fail, but we fail to do so.

Duplicate the logic we have for the download helper: if the post-process
helper fails, remove the downloaded stuff, and continue on to the next
URI, which will ultimately hit the backup mirror (if one has been
configured).

Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-12 09:38:54 +01:00
Peter Korsgaard
5be2d9bb62 support/dependencies/dependencies.sh: require diagnostics perl package for mosh
Fixes:
http://autobuild.buildroot.net/results/820/820e98b1c126469b1f180f078d102ded43b9c40e/

scripts/Makefile.am of mosh-1.4.0 needs the perl diagnostics module on the host:

make[3]: Entering directory '/home/buildroot/autobuild/instance-2/output-1/build/mosh-1.4.0/scripts'
perl -Mdiagnostics -c ./mosh.pl
Can't locate diagnostics.pm in @INC (you may need to install the diagnostics module) (@INC contains: /home/buildroot/autobuild/instance-2/output-1/host/lib/perl /usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5).
BEGIN failed--compilation aborted.

So add a check for it in dependencies.sh similar to the other perl modules.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-12 09:28:45 +01:00
Thomas Petazzoni
f25b0b6d1d support/testing/tests: add NodeJS tests
This commit adds two new test cases:

 - TestNodeJSBasic which builds a target configuration with just
   NodeJS enabled, and which runs a very simple NodeJS script on the
   target.

 - TestNodeJSModule, which builds a target configuration with NodeJS
   enabled + the installation of one extra module, which means npm on
   the host (from host-nodejs) is used, and which runs a very simple
   NodeJS script on the target that uses this extra module.

Having both tests separately allows to validate that both nodejs-only
and nodejs+host-nodejs configurations behave correctly, at least in
minimal scenarios.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-11 00:13:08 +01:00
Julien Olivain
a82eec628c support/testing/tests/package/test_gnupg2.py: new runtime test
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-10 23:36:16 +01:00
Ricardo Martincoski
c5a3dfb9a4 utils/check-package: check all shell scripts
Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).

Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.

In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.

Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.

Apply these checks for shell scripts:
 - shellcheck;
 - trailing space;
 - consecutive empty lines;
 - empty last line on file;
 - newline at end of file.

Update the list of ignored warnings.

Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 13:31:06 +01:00
Peter Korsgaard
f229564c36 support/dependencies/dependencies.sh: ensure git is >= 2.0.0 for vendoring
The go vendoring fails on CentOS 7 (which uses git 1.8.3.1) with errors
related to shallow clones:

make docker-compose-source
..
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4
github.com/docker/compose/v2/pkg/mocks imports
        github.com/theupdateframework/notary/client imports
        github.com/docker/go/canonical/json: github.com/docker/go@v1.5.1-1.0.20160303222718-d30aec9fd63c: invalid pseudo-version: git fetch --unshallow -f origin in /home/jacmet/source/buildroot-mirror/output/host/share/go-path/pkg/mod/cache/vcs/48fbd2dfabec81f4c93170677bfc89087d4bec07a2d08f6ca5ce3d17962677ee: exit status 128:
        fatal: git fetch-pack: expected shallow list
make[1]: *** [/home/jacmet/source/buildroot-mirror/output/build/docker-compose-2.15.1/.stamp_downloaded] Error 1

It works with git 2.0.0 (released May 2014, included in Debian 8), so check
for >= 2.0.0 with logic similar to the GNU patch version check.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 23:03:10 +01:00
Yann E. MORIN
a016b693f7 gitlab-ci: fix pipelines
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.

We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore
the situation.

It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.

The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).

Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).

[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:55:49 +01:00
Yann E. MORIN
e92bb8f1c9 gitlab-ci: don't use before_script in job templates
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.

We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.

Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.

There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:55:49 +01:00
Peter Korsgaard
4fbd2f6d2c support/dependencies/dependencies.sh: silence gcc plugin test
The gcc plugin test was not using the -q option to grep causing it to print
the line to stdout, so fix that.

While we're at it, adjust the locale check to use grep -q instead of
redirecting to /dev/null for consistency with the other checks.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-07 18:24:10 +01:00
Thomas Petazzoni
7a7c6725f6 support/docker: use APT::Retries in apt-get calls to retry
snapshot.debian.org is notoriously slow, and quite often leads to
timeouts when downloading packages. To give us a better chance of
succeeding, let's retry 3 times before failing.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-07 18:14:59 +01:00
Thomas Petazzoni
d24c6f4b15 support/docker: update Debian distro to a newer tag/date
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-07 18:14:59 +01:00
Julien Olivain
4e23807372 support/testing/tests/package/test_python_magic_wormhole.py: new runtime test
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 14:01:05 +01:00
Julien Olivain
9aaef2a077 package/python-spake2: new package
SPAKE2 password-authenticated key exchange (in pure python).

This library implements the SPAKE2 password-authenticated key
exchange ("PAKE") algorithm. This allows two parties, who share a
weak password, to safely derive a strong shared secret (and
therefore build an encrypted+authenticated channel).

https://github.com/warner/python-spake2

Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 13:41:48 +01:00
Julien Olivain
433ce2966f package/python-hkdf: new package
HMAC-based Extract-and-Expand Key Derivation Function (HKDF).

https://github.com/casebeer/python-hkdf

Signed-off-by: Julien Olivain <ju.o@free.fr>
[Thomas: justify the license.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 12:50:27 +01:00
Peter Korsgaard
b63e155e5f support/dependencies/dependencies.sh: correct check for open perl module
Commit 4cdd99190e (support/dependencies/dependencies.sh: require open perl
package for libxcrypt) added a check for the "open" perl module for
libxcrypt, but it does not work as "open" cannot be directly used with
"require" as an argument is needed:

perl -e "require open"
Not enough arguments for open at -e line 1, at EOF
Execution of -e aborted due to compilation errors.

So special case the check to instead check with "use open ':std'".

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-07 12:35:46 +01:00
Peter Korsgaard
4cdd99190e support/dependencies/dependencies.sh: require open perl package for libxcrypt
Fixes:
http://autobuild.buildroot.net/results/16c/16cba783be11cc5024f9c56bb0a7abb0acf13ef8/

The configure script of libxcrypt needs the open perl module on the host:

Can't locate open.pm in @INC (you may need to install the open module) (@INC
contains: /home/buildroot/autobuild/instance-2/output-1/host/lib/perl
/usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5
/usr/share/perl5) at ./build-aux/scripts/expand-selected-hashes line 20.

So add a check for it in dependencies.sh similar to how it is done for mpv.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-07 10:33:33 +01:00
Peter Korsgaard
63da115085 support/testing: test_check_package.py: fix flake8 line length error
Commit d631615eb1 (support/testing: test check-package ignore list) added
a too long line causing the check-flake8 target to fail:

support/testing/tests/utils/test_check_package.py:233:133:
E501 line too long (138 > 132 characters)

https://gitlab.com/buildroot.org/buildroot/-/jobs/3726245521

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-07 10:06:28 +01:00
Peter Korsgaard
8f8085e63a support/dependencies/dependencies.sh: require English perl package for host-libxml-parser-perl
Fixes http://autobuild.buildroot.net/results/166/1667b4ebd8e16e7d8b47bc3ca128a76daf27b4d9/

The configure script of host-libxml-parser-perl needs the English perl
module on the host:

ERROR from evaluation of
/home/buildroot/autobuild/instance-1/output-1/build/host-libxml-parser-perl-2.46/Expat/Makefile.PL:
Can't locate English.pm in @INC (you may need to install the English module)
(@INC contains:
/home/buildroot/autobuild/instance-1/output-1/build/host-libxml-parser-perl-2.46/inc
/home/buildroot/autobuild/instance-1/output-1/host/lib/perl
/usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5
/usr/share/perl5 .) at ./Makefile.PL line 3.

So add a check for it in dependencies.sh similar to the existing check for
ExtUtils::MakeMaker.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-07 09:39:29 +01:00
Yann E. MORIN
82f84b4cfb support/tests: enable PPD, and thus TLPB, for systemd tests
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:34:08 +01:00
Yann E. MORIN
2c41fb6225 support/tests: allow top-level parallel builds
Running tests with top-level parallel builds can speed up running some
tests, expecially those that have a lot of packages like the systemd
init tests.

Trigger TLPB when the configuration enables per-package directories.

We're using the jlevel argument, which normally is used for BR2_JLEVEL
as the value for calling make -j<N> at the top-level. In fact,
BR2_JLEVEL is "unused" when using TLPB, because the top-level make
acts as the job server that distributes tokens to sub-makes (except
for the few build systems like waf or scons that don't support this),
so it's really the top-level make -j<N> that determines the level of
parallelism, and BR2_JLEVEL doesn't really have an effect.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Thomas: extend explanation a bit]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:31:41 +01:00
Ricardo Martincoski
de03312db0 utils/get-developers: print error for correct line
Start counting the line numbers in 1 instead of 0, in case an error
must be printed.

Both the error about a developer entry with no file entry and the error
about a file entry with no developer entry actually belong to the
non-empty line previous the one being analysed, so in these cases print
the line number from the line before.

Also count empty and comment lines, so a developer fixing the file can
jump to the correct line (or the nearest one).

At same time standardize the messages, printing the line number
also in the case of a warning for a file that is not in the tree
anymore.

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:24:16 +01:00
Ricardo Martincoski
5ee1dd85a3 utils/get-developers: bail out on parsing errors
Currently 4 types of parsing errors/warnings can be found:
- entry for a file that is not in the tree anymore (warning)
- developer entry with no file entry (error)
- file entry with no developer (error)
- entry that is not a developer, a file or a comment (hard error)

Currently only the last one ends the script with -v with error code.

Make all 3 error types into hard errors and bail out at the first error
found, because the rest of the state machine is not designed to handle
malformed input.
Suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:24:16 +01:00
Yann E. MORIN
557c68fbd2 support/tests/cpio: extend runtime tests for extra dract modules
Add a basic module that installs a single file, to check that it is
properly accounted for.

Reported-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thierry Bultel <thierry.bultel@linatsea.fr>
Cc: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 22:46:35 +01:00
Peter Korsgaard
5aab8c59d5 support/dependencies/dependencies.sh: require FindBin perl package for libopenssl
Fixes:
http://autobuild.buildroot.net/results/5d1/5d10f4f545dccf126e3f5b5efce777a393c5e7bc/
http://autobuild.buildroot.net/results/7c2/7c2df31c6eb34b68e460e092d8c262bdc6c8f25a/

The configure script of (host-)libopenssl needs the FindBin perl module on the host:

Can't locate FindBin.pm in @INC (you may need to install the FindBin module)
(@INC contains: /home/buildroot/autobuild/instance-2/output-1/host/lib/perl
/usr/local/lib64/perl5/5.36 /usr/local/share/perl5/5.36
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5
/usr/share/perl5) at ./Configure line 15.  BEGIN failed--compilation aborted
at ./Configure line 15.

As this is needed for both libopenssl and host-libopenssl (which does not
have a corresponding config symbol in the .config), we have to require it
unconditionally.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 22:39:15 +01:00
Thomas Petazzoni
452365a385 toolchain/toolchain-external/toolchain-external-bootlin: update to 2022.08
This commit is the result of running
support/scripts/gen-bootlin-toolchains now that 2022.08 toolchains
have been made available.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 22:26:24 +01:00
Ricardo Martincoski
bcd15aa08a support/docker: add python3-magic
It will be needed by check-package to run checks according to the file
type (the same determined by the command 'file').

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 18:54:18 +01:00
Ricardo Martincoski
d631615eb1 support/testing: test check-package ignore list
Extend test_check_package to also check the ignore list functionality.
Check:
- the entries in the ignore list use relative path;
- an entry in the ignore list actually ignores the warning;
- an outdated entry in the ignore list generates a warning by its own,
  preventing the ignoring list to grow indefinitely.

For this to work, add 3 test fixtures, listing entries for an
pre-existing file in the br2-external used in the test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 18:54:18 +01:00
Ricardo Martincoski
0c5472ace2 utils/check-symbols: new script
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .

The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).

Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.

At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
  (for legacy handling) but is referenced in the package without a
  comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
  but it is not actually referenced.

There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.

Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'

At same time the script is created:
- add unit tests for it, they can be run using:
  utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
  using the script) and check-check-symbols (to check the script against
  its unit tests)

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
	change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 16:30:20 +01:00
Ricardo Martincoski
0ad3ae14c5 support/scripts/generate-gitlab-ci-yml: improve test-pkg support
Following the example of test-pkg config described in commit
"12c7a05da1 utils/test-pkg: add gitlab-ci support" to test a defconfig
fragment that contains a disabled option is currently possible, but
it do requires one to change the git config core.commentChart so the
lines starting with "#" are not discarded by git when creating/editing
the commit message.

For instance, without the indentation the 3rd line below would be
excluded from the commit message when the editor is closed:
    test-pkg config:
    SOME_OPTION=y
    # OTHER_OPTION is not set
    SOME_VARIABLE="some value"

Requiring to change git configs is not very nice.
So make the developer's life easier by changing the sed expression to
remove indentation with spaces from a defconfig fragment found on a
commit message.
For instance these lines become valid and generate a defconfig fragment
without the indentation of one space to be tested in GitLab CI:
test-pkg config:
 SOME_OPTION=y
 # OTHER_OPTION is not set
 SOME_VARIABLE="some value"

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 14:22:37 +01:00
Yann E. MORIN
93e7fc3e53 toolchain: make paranoid check of library/header paths unconditional
When we introduced support for the paranoid check of unsafe libraries
and headers path with commit 4ac8f78d37 (Add option for paranoid
unsafe path checking) back in 2014, we made it optional, as we expected
that would break quite a few packages.

Now, almost 8 years later, we only have three packages that explicitly
reference the option (dillo, gnuradio, and libtalloc), either in a patch
or in their .mk.

The option has been enabled by default since 2016, with 61c8854cef
(toolchain: enable paranoid unsafe path check by default), and that has
not triggered many build failures in a while.

The minimal defconfig used by test-pkg has also had it enabled as of
b6c98b3549 (minimal.config: add BR2_COMPILER_PARANOID_UNSAFE_PATH=y)
in 2017.

It is time to make that globally unconditional now.

There is still a remnant, in our binutils patches. As our toolchain may
get used outside of Buildroot, people may got the expectation that path
poisoning is only a warning, so we keep the current behaviour.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Acked-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-05 15:11:25 +01:00
Julien Olivain
d4ecefa3cc package/perftest: new package
Open Fabrics Performance Tests.

This is a collection of tests written over uverbs intended
for use as a performance micro-benchmark. The tests may be
used for HW or SW tuning as well as for functional testing.

https://github.com/linux-rdma/perftest

Tested-by: Shamraiz Ashraf <shamraizashraf092@gmail.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
[yann.morin.1998@free.fr
  - move all arch-related dependencies to _ARCH_SUPPORTS
  - include musl condition in comment and its dependnecies
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-05 15:03:50 +01:00
Julien Olivain
ea47e177f0 package/rdma-core: new package
This is the userspace components for the Linux Kernel's
drivers/infiniband subsystem.

https://github.com/linux-rdma/rdma-core

Tested-by: Shamraiz Ashraf <shamraizashraf092@gmail.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
[yann.morin.1998@free.fr:
  - select iproute2 as it provides the 'rdma' utility
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-05 13:51:13 +01:00
Peter Korsgaard
a796cfb2e0 Update for 2022.11.1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit d48a8beb39)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-01-18 17:50:12 +01:00
Yann E. MORIN
fbddd12601 support/download: fix concurrent cargo vendor
Commit 8450b76918 (package/pkg-cargo: move CARGO_HOME into DL_DIR)
allowed for a shared cargo cache of crates. Internally, cargo is
supposed to lock themselves when accessing that cache, and that commit
even had some research in that area, pointing at [0] for complaints
about too-coarse the lock, so it was deemed safe to have a shared cargo
home.

However, in practice, the locking as implemented by cargo, fails to
properly protect the concurrent accesses to the crates cache, with random
failures that manifest themselves like so:

        Blocking waiting for file lock on package cache
        Blocking waiting for file lock on package cache
     Downloading crates ...
    error: failed to sync
    Caused by:
      failed to download packages
    Caused by:
      failed to download `autocfg v1.1.0`
    Caused by:
      unable to get packages from source
    Caused by:
      failed to unpack package `autocfg v1.1.0`
    Caused by:
      failed to unpack entry at `autocfg-1.1.0/src/tests.rs`
    Caused by:
      No such file or directory (os error 2) while canonicalizing [...]

with the last few errors sometime being:

    Caused by:
      failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml`
    Caused by:
      can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path

So, as we do not systematically use our own cargo build (we can use a
pre-built one with host-rust-bin), we can't patch cargo (even if we knew
what to do!).

Instead, we implement a lock ourselves, by wrapping the call to "cargo
vendor" with a flock(1) on cargo home.

Note: the download wrapper is already flock-ed, but it is a per-package
lock, so it does not prevent different packages from being downloaded in
parallel; if those packages need cargo vendoring, that will not be
protected by the flock on the dl wrapper. So we really do need a flock
on cargo home.

[0] https://github.com/rust-lang/cargo/issues/6930

Fixes: 8450b76918

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Moritz Bitsch <moritz@h6t.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-01-14 21:00:16 +01:00
Yann E. MORIN
ac2b26fba2 support/download: fix cargo vendoring
Commit de4cf25375 (package/{rust, rust-bin}: bump to version 1.66.0)
forgot, despite the big comment above the version strings, to confirm
that the vendoring was still working.

Previously, we were adding the vendoring equivalence manually, but in
commit 04154a6517 (support/download/cargo-post-process: cargo output
for vendor config), we switched to using the output of "cargo vendor"
(on stdout) to support cases were the vendoring equivalence would be
more complex (e.g. when using crates not hosted on crates.io).

With rust until and including 1.65.0, "cargo vendor" would output (for
crates.io crates) the same output as our manual fixups, except it was
preceded by an empty line. So, to avoid recompting all our hashes, we
added a tweak to strip away the leading empty line in 04154a6517.

But rust 1.66.0 includes [0] which changes the output (on stdout) of
"cargo vendor", where the first empty line is no longer emitted.

This means that our tweak for rust 1.65.0 now strips out an important
part of the cargo vendor output, which renders the archives invalid, and
thus generates different archives, which fail to validate against our
hashes.

Fix this by doing what the comment in the post-process helper states,
and just keep the whole output of "cargo vendor", by just removing the
"tail --lines=+2". Since that comment is no longer meaningful, we drop
it too.

Now, all our 6 cargo-based packages, as well as our 5 python packages
that have rust code, can be vendored again, without changing our hashes,
but most importantly, with valid archives.

Still, we keep the comment above the versions strings, in the hope that
a future bumper will notice and be more careful at validating the
vendoring.

[0] https://github.com/rust-lang/cargo/pull/11273

Fixes:
    http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/  (bat)
    http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/  (dust)
    http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/  (hyerfine)
    http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/  (ripgrep)
    ... and so many others...

Reported-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: James Hilliard <james.hilliard1@gmail.com>
Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Reviewed-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-01-10 17:58:05 +01:00
Marcin Niestroj
82522d2547 package/python-dbus-fast: new package
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Reviewed-by: James Hilliard <james.hilliard1@gmail.com>
[yann.morin.1998@free.fr:
  - select host-cython in Config.in
  - introduce obj_path in test sample
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-01-02 22:33:55 +01:00
Yegor Yefremov
2f0a2a43bb support/testing: add test for python-munch
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-12-29 10:38:11 +01:00
Yann E. MORIN
44161560dd support/tests: print failed command and output on assertRunOK error
Currently, when asserting that a command succeeded, we just capture the
return code of the command. If that is not zero, the assertion fails,
but the error message is not very splicit:
    AssertionError: 1 != 0

Replace the error message with an explicit message that dumps the failed
command, the error code, and the resulting output.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-12-27 21:40:31 +01:00
Yann E. MORIN
0627cb0e8a support/testing: remove leftover kernel config fragment
Commit 86d32208b6 (support/testing/tests/init/test_systemd.py: use
downloaded kernel) stopped building a custom kernel for the systemd
tests, but forgot to drop the associated kernel config fragment.

That fragment is now not used in any test case, so we can drop it.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-12-22 11:47:26 +01:00
Yegor Yefremov
b54e0dde30 support/testing: add test for python-dicttoxml2
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-12-11 21:18:57 +01:00
Julien Olivain
6ad775fc81 support/testing/tests/package/test_ncdu.py: new runtime test
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-12-11 11:50:15 +01:00
Peter Korsgaard
6ca0edcb5f Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2022-12-05 10:01:26 +01:00
Peter Korsgaard
40bd4a32aa Update for 2022.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2022-12-05 08:23:19 +01:00
Julien Olivain
2ad68ff8df package/z3: new package
Z3, also known as the Z3 Theorem Prover, is a cross-platform
satisfiability modulo theories (SMT) solver.

https://github.com/Z3Prover/z3

Signed-off-by: Julien Olivain <ju.o@free.fr>
[yann.morin.1998@free.fr:
  - python bindings 'depends on' python, not 'select' it
  - fix check-package in test_z3.py
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-11-20 14:54:58 +01:00
Peter Korsgaard
12a33b54b3 Update for 2022.08.2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 25680e6aa8)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2022-11-17 08:24:23 +01:00
Gleb Mazovetskiy
f7a6011f78 Makefile: unexport PERL_MM_OPT
Compilation of Perl-related packages fails if `PERL_MM_OPT` is defined.

We previously issued an error in this case.
Instead, simply `unexport` the variable.

Signed-off-by: Gleb Mazovetskiy <glex.spb@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-07 23:10:47 +01:00
Thomas Petazzoni
b9dd9ee857 support/testing/tests/package/test_gdb.py: drop version-specific tests
Back when support/testing/tests/package/test_gdb was introduced, there
was a significant difference in how gdb < 10 and gdb >= 10 were
handled in gdb.mk, which explained why we were testing both gdb 9.x
and gdb 11.x.

However, support for gdb 9.x has now been dropped, and we only support
gdb >= 10.x, so testing gdb 9.x and 11.x separately no longer make
much sense. In addition:

 - other GDB tests in the same file already test the default version,
 which is now 11.x, meaning we in fact have duplicated tests between
 the ones testing the default version and the ones testing 11.x
 specifically

 - GDB 9.x has been removed, which means all the tests testing GDB 9.x
 are failing, with a Config.in.legacy build error.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/3249828456 (TestGdbHostOnly9x)
  https://gitlab.com/buildroot.org/buildroot/-/jobs/3249828454 (TestGdbHostGdbserver9x)
  https://gitlab.com/buildroot.org/buildroot/-/jobs/3249828451 (TestGdbHostGdbTarget9x)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-03 23:08:10 +01:00
Thomas Petazzoni
e7930708a3 support/testing/tests/package/test_python_crossbar: use ext2 instead of cpio
The CPIO filesystem generated by the test_python_crossbar test is too
large, and doesn't fit as an initramfs in the 256MB of RAM available
in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root
filesystem, ultimately causing the test to fail.

It would make sense to switch all test cases to use ext2 + a
hard-drive, but for now, let's fix the few test cases that are causing
problems.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/3249828587

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-03 23:07:58 +01:00
Thomas Petazzoni
3884cfc057 support/testing/tests/package/test_python_flask*: increase time after server startup
It seems like on Gitlab CI, the runners are quite slow, and the Flask
server does not startup in the 15 seconds we give it. So increase this
to 30 seconds before trying to contact the Flask server.

Hopefully fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/3249828594

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-03 23:07:54 +01:00
Christian Stewart
5d8371a3fe support/testing/tests/package/test_docker-compose: update kernel to 4.19.262
Signed-off-by: Christian Stewart <christian@paral.in>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-01 19:15:01 +01:00