The python-magic-wormhole runtime test can randomly fail on slow
runners, see [1].
The issue is that the sending command is started first in background
_without_ redirecting its output to /dev/null. The receiving command
is started after, expecting the message to be printed on the first
standard output line. On slower systems, the sending command still
print messages while the test controller expect output from the
receiving command. The expected string finally appear, but not on the
first line. This makes the test fail.
This commit fixes the issue by redirecting all outputs (stdout, stderr)
of the sending command to /dev/null. To help even more, the sleep time
is moved from the emulator to the test controller. The sleep time is
also multiplied by the timeout_multiplier.
Fixes: [1]
[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/6888691508
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
The python-hid package got tweaked when applied in commit 4432b08790
(package/python-hid: new package) so it now selects
BR2_PACKAGE_HIDAPI, and we therefore do not need to explicitly enable
hidapi in the test.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Our git download backend switches the remote of our local clone, and
fetches all refs and tags from that remote.
When the local clone has a tag fetched from another remote, and the new
remote also has a tag by the same name, and that tag points to another
commit, then git refuses to fetch the new tag and exits in error, as it
considers that the new tag would clobber the existing one. This is safe
and sane behaviour when run interactively with a human that can take a
decision.
However, in our case, we don't care about any tags that were present
before, as only the last one makes sense in our case: the one from the
remote the user has requested for the current build.
Tell git to forcefully pull tags, even if they would clobber existing
ones.
Note that, although this changes the git backend, it does not change the
content of generated archives, so we do not need to bump the suffix
version.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
We can't stay in the past forever and ever...
Since tar 1.35, the way some fields (devmajor and devminor) are stored
has changed. These fields exist for each file in the tarball, but only
used for device nodes. In previous versions of GNU tar, they were set to
zero; since 1.35, they are set to empty.
Although this doesn't change anything about the content of the tarball,
and it will be extracted in exactly the same way regardless of the tar
version used for extracting, it does change the hash of the tarball.
Therefore, we have to
- make sure that the correct version of tar is used;
- update the format version so that the filename is different from
before.
Increment all BR_FMT_VERSION by one.
Require tar >= 1.35 instead of < 1.35.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Arnout: also increment BR_FMT_VERSION and extend the commit message]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Replace the names of the tarballs in the hash files to -git3.
We don't have any symlinks in the tests, so the hashes themselves don't
change.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Currently, when we generate archives, we rely on a few assumptions and
mechanisms to ensure reproducilibity. So far, we mostly accounted for
the content (i.e. content, filenames, and path) of the files we
archived, and this is OK (git and svn should provide reproducilbe
content by design, and cargo and go vendoring are also supposed to be
generating reproducible content.
However, tarballs do not only contain the content of the files; they
also have a few metadata about those files. Beyond filenames and paths,
which are already reproducible, there is the timestamp, the user and
group name and ID. Those are also accounted for and made reproducible.
The final touch (so far!) is that files have access rights (aka mode),
and those too are stored in tarballs. So far we accounted for those by
ensuring that Buildroot would always run under a known umask, thus
generating files with reproducible modes.
That falls short in one case that we did not envision, though: a shared
download directory, where extended attributes are set to provide a
default ACL that is permissive, to allow two or more users (with
different uid and gid) to all read and write to such a directory. This
is trivially achieved with something like:
$ mkdir -p "${BR2_DL_DIR}"
$ setfacl -m 'default:user::rwx' "${BR2_DL_DIR}"
$ setfacl -m 'default:group::rwx' "${BR2_DL_DIR}"
$ setfacl -m 'default:other::rwx' "${BR2_DL_DIR}"
This has the effect that:
- files below BR2_DL_DIR are all set with user, group, and world read
and write access,
- files executable by the owner will also be group and world
executable,
- directories are user, group, and world readable, writable, and
searchable.
This means that all the archives we generate from files in BR2_DL_DIR
will have modes that are different from those generated on other systems,
where only the traditional umask is used.
There are various solutions to solve that issue:
- detect the situation and abort: that's not nice, because users have
a legitimiate reason to want to share that directory,
- find a solution for each affected download mechanism: git, svn, hg,
cvs, bzr... and for each of the affected vendoring mechanism: go and
cargo [0]; this is not nice, because it means a lot of repetition,
with the risk that they diverge over time (e.g. one is fixed for a
newer issue, while the others are left out due to an oversight...)
- find a single, common solution that works in all cases, whatever the
download mechanism and/or vendoring: this is the best, because we
can extend and fix it once and everything else benefits from it.
We obviously go for the third option.
The common solution is rather simple. When creating the tarball in
support/download/helpers, give an option to tar to set the group and
other permissions to those of the user, but without write permission.
This implies that we must bump the version-suffix for the download
backends [1] and for the vendoring post-processes. It also implies that
the hash may change, under the following circumstances:
- Symlinks normally have permissions 0777 (because symlink permissions
are in fact meaningless). They will now have permission 0755 in the
tarball.
- If the original tarball (for vendored go and cargo packages) contained
files that are readable or executable by owner but not by group or
other, they will now be readable resp. executable by group and other
too. Note that for writeable it is not the case, because those were
already handled by our 0022 umask (which makes them not writeable by
group and other).
Because the hash may change, we need to update the BR_FMT_VERSION for
everything that creates tarballs. Go and cargo didn't have one up to
now, the the previous commit added the possibility to give one. The ones
for git and svn have to be updated. Since it is now possible to have a
suffix for both the VCS and the post-processing, change the suffix to
something more descriptive than "-brX", i.e. -git3 for git, -go1 for
golang, etc.
The hash updates and filename changes will be handled in a follow-up
commit.
[0] Note however that the vendoring is currently not done in a
sub-directory of BR2_DL_DIR, but the cargo and go caches are located
there. Files that get copied from there to the vendoring area would be
tainted as well, and thus we want to address that situation as well.
[1] we currently do not have a CVS version suffix, because we do not
guarantee the reproducilibity of CVS archives (we can't); for hg, we are
currently using hg's own archive tool, and presumably that does not have
the mode issue because it is not using the checked-out files. Still,
doing the mode fix in a single location will help extend those two
backends in the future (if that ever happens...).
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
The version suffix for the git-generated archives has changed, so update
the filenames accordingly in the hash files. The content of the archives
has not changed, though.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Files in a git repository can be given attributes, like the usual eol
that can convert to-from crlf, cr, lf; those are applied when comitting
or checking-out a file.
There are also two attributes that are meant to be used when generating
an archive (with git archive): export-subst, and export-ignore, that
respectively substitutes format placeholders in a file, and excludes a
file from the archive.
Some package (e.g. pcm-tools, luajit) use the export-subst attribute
to generate versioning information. luajit, specifically, uses the UNIX
timestamp of the commit as the patch-level for its semantic versioning.
We don't use git-archive, because we need to get submodules and LFS
blob, which git-archive does not handle. So, our git backend tries to
impersonate git-archive as much as possible, but the support for git
attributes was lost when we converted it from using git-archive to
manually creating the tarball in 3abd5ba424 (support/download/git: do
not use git archive, handle it manually) in preparation for f109e7eeb5
(support/download/git: add support for submodules) (arguably, a long
time ago...)
Extend the git backend to handle the export-subst attribute. There is
no git tool (that we could find) that does that automatically, except
git-archive, which we can't use; "git check-attr" however can report
whether a file has a specific attribute (and git check-attr can work
with \0-delimited fields and records).
So, we iterate over all the files in the repository, and filter those
that have the export-subst attribute set. Then for each file, we use a
bit of awk to do the replacement:
- for each line (managed natively by awk), we iterate over each
format placeholder,
- for each placeholer, we query "git log" with the requested format,
- we emit the replacement.
When doing the replacement, we decided to force abbreviating short
hashes to 40 chars, which is the length of a full sha1, rather than
actually abbreviating them:
- letting git decide of the length is not reproducible over time:
- as new commits are added, the short length will increase to avoid
collisions,
- newer git versions may decide on a different heuristic to shorten
hashes,
- users may have local settings with an arbitrary length (in their
~/.gitconfig for example);
- deciding on our side of an "small" arbitrary value would not be
viable long term either, as it might be too large to be minimum, or
too short to avoid collisions.
The only reproducible solution is to use unabbreviated hashes.
Handling git-attributes also implies that the format of the generated
archives has changed, since we now expand placeholders, so we bump our
git format version.
Hash files for all git-downloaded packages will be updated in followup
commits.
Of all our git-downloaded packages, 5 are affected, and their hashes
will be updated in a followup commit too:
- pcm-tools, which was known, and the one that triggered this commit;
since we now expand placeholders, we can drop the post-extract hook;
switching to a full hash in replacements also changes the hash of
the generated archive;
- qt5knx, qt5location, qt5mqtt, and qt5opcua: the file .tag at the
repository root, contains only the full hash placeholder; that file
is not used at all during the build (AFAICS);
Finally, a sixth package, luajit, uses export-subst; it currently relies
on the github-generated archive (because it happens to currently use a
format that is reproducible); it will also be converted in a follow-up
patch.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Woody Douglass <wdouglass@carnegierobotics.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Francois Perrad <fperrad@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
SymPy is a Python library for symbolic mathematics. It aims
to become a full-featured computer algebra system (CAS)
while keeping the code as simple as possible in order to be
comprehensible and easily extensible. SymPy is written
entirely in Python.
https://www.sympy.org/
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
mpmath is a free (BSD licensed) Python library for real and
complex floating-point arithmetic with arbitrary precision.
https://mpmath.org/
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The lvm2 package now enables the required Kernel configuration. The
Kernel config fragment included in this test is no longer needed.
This commit removes it.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 0155095c55)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The CPIO filesystem generated by TestPerlDBDmysql is too large, and
doesn't fit as an initramfs in the 256MB of RAM available in the
versatilepb machine. This causes a failure while running a basic
test "perl -MDBI -e '1'" since "/usr/lib/perl5", and many files
being missing from the root filesystem, ultimately causing the test
to fail.
Can't locate DBI.pm in @INC (you may need to install the DBI module)
(@INC entries checked: /usr/lib/perl5/site_perl/5.38.2/arm-linux
/usr/lib/perl5/site_perl/5.38.2 /usr/lib/perl5/5.38.2/arm-linux
/usr/lib/perl5/5.38.2).
It would make sense to switch all test cases to use ext2 + a
hard-drive, but for now, let's fix the few test cases that are causing
problems.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6735654506
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit bumps to the 6.6.x series to refresh the test, as the
4.19.x series is planned to be end-of-life soon, in December 2024.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The Sourcery CodeBench Nios-II 2018.05 external toolchain will be
removed, first remove it from toolchain-configs.csv used by test-pkg
and genrandconfig scripts.
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Since the toolchain Bootlin update to 2023.11-1 [1], the arm Linux
kernel build is broken with binutils >= 2.41 with:
arch/arm/mm/proc-v7.S: Assembler messages:
arch/arm/mm/proc-v7.S:640: Error: junk at end of line, first unrecognized character is `#'
A similar issue has already be fixed for qemu m68k [2].
Bump to the latest kernel 4.19 that already include the backport
of 790756c7e022 ("ARM: 8933/1: replace Sun/Solaris style flag on section directive")
[1] 7e0e6e3b86
[2] a1ce9474e4
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6703222383
Signed-off-by: Romain Naour <romain.naour@smile.fr>
GCC14 now treats implicit int types as error so when check() from
check-lxdialog.sh is called to check whether we can link against ncurses
it will fail silently and the help text indicating to install ncurses is
printed.
However, this is not due to missing ncurses but once the stderr redirect
to /dev/null is removed we can see the root cause:
<stdin>:2:1: error: return type defaults to ‘int’ [-Wimplicit-int]
So, in order for menuconfig to work with GCC14 lets just specify the
return type of main() as int.
Npte that the upstream kconfig in the linux kernel source tree no longer
carries or uses the check-lxdialog.sh script since commit 1c5af5cf9308
(kconfig: refactor ncurses package checks for building mconf and nconf),
so there is no commit we can backport to our kconfig copy.
Signed-off-by: Robert Marko <robimarko@gmail.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Tested-by: Petr Vorel <petr.vorel@gmail.com>
[yann.morin.1998@free.fr: add note about upstream kernel]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Also a runtime testcase is added.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[yann.morin.1998@free.fr:
- fix check-package
- small codign style in test sample
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
TestATFVexpress is using vexpress_aemv8a_juno as as u-boot defconfig
but the Buildroot defconfig of this board was removed in 2022.11 [1]
Since both TestATFVexpress and TestATFAllwinner are now using mainline
ATF, we don't really need several ATF test anymore. Initially [2],
several runtime test were added to test ATF/U-Boot combinations when
ATF was provided by a vendor: vexpress (mainline), Allwinner and
Marvell.
Keep TestATFAllwinner as ATF mainline test.
[1] 347c108738
[2] 8cf3ce04e9
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
u-boot-2021.04 seems to be broken when pylibfdt support is enabled
and the latest python3/setuptools are used.
Since the TestATFAllwinner is using bananapi_m64 as u-boot defconfig
but the Buildroot defconfig of this board was removed in 2022.11 [1]
update TestATFAllwinner to use a newer BSP. Use the one provided
by orangepi_zero_plus2_defconfig.
[1] daf3c6661f
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6477656317 (TestATFAllwinner)
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit ccb4e5db5c ("utils/check-package: emit library name along with check function name")
updated the .checkpackageignore format but forgot to update
.checkpackageignore files used in TestCheckPackage.
Keep .checkpackageignore_outdated as is since it must be outdated.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6568786785
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Since the toolchain Bootlin update to 2023.11-1 [1], the arm Linux
kernel build is broken with binutils >= 2.41 with:
arch/arm/mm/proc-v7.S: Assembler messages:
arch/arm/mm/proc-v7.S:640: Error: junk at end of line, first unrecognized character is `#'
A similar issue has already be fixed for qemu m68k [2].
Bump to the latest kernel 4.19 that already include the backport
of 790756c7e022 ("ARM: 8933/1: replace Sun/Solaris style flag on section directive")
[1] 7e0e6e3b86
[2] a1ce9474e4
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6414160106 (TestFileCapabilities)
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
sortGrid()'s handling of git hashes and other large hex numbers
has been inconsistent, they can be detected as strings or numbers
depending on what type of character they start with.
This patch fixes the behaviour by using a regex to capture everything
that looks like a big hex number and treat it as a string.
This means when you sort by current version ascending all the version
strings with big hex numbers should show up first, sorted 0-9,a-f.
First we check for a string length >= 39, and then apply a regex
to return an array with every char from that string that matched
the regex. If the length of this array is still >= 39 we can assume
we are looking at something containing a git hash.
The reason why the length is defined as ">= 39" and not "40" or
"39 or 40" is twofold:
Firstly, 39 was chosen as a minimum to match stuff with 39 char git
hashes, like the rockchip-mali package.
Secondly, there is no max because we actually want to catch not
just explicitly git hashes, but any verson string with big gnarly
hex numbers in it.
Stuff like: "1.4.2-168-ged3039cdbeeb28fc0011c3585d8f7dfb91038292"
Why? Well, the idea is less about git hashes and sorting
and more about grouping similarly formatted version strings.
It would be impossble (or at least annoyingly complicated) and of
dubious utility to get a real sequential sort out of the
current version column, so the attempt here is to at the very
least collect all the similarly formatted things together.
This isn't perfect, but it's a (arguably) more useful sorted
output than before.
A demo is available here:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-improve-git-hash-sorting.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Because the div_class variable was not reassigned a value,
cells in the latest_version column were still being assigned
hash_file classes and so were being picked up as elements in
the hash_file sort. This lead to execessive grid gap elements
stacking up and creating huge blank spaces at the top of the page.
This is very noticable on pages with a large number of packages,
like the ones the autobuilder creates.
original behaviour(click the "Hash file" column label twice):
http://autobuild.buildroot.org/stats/master.html
demo of fixed behaviour:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-bug-when-sorting-by-hash-file.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024.02 Bootlin toolchains have been released, so let's update the
support in Buildroot. Notable changes:
- Bleeding edge toolchains now use binutils 2.42, and stable
toolchains use binutils 2.41. This fixes binutils bug 27597
for both.
- glibc has been updated to 2.39
- musl has been updated to 1.2.5, which brings 32-bit RISC-V
support. Due to this, 2 new toolchain variants are added: 32-bit
RISC-V stable, 32-bit RISC-V bleeding edge.
- Bleeding edge toolchains now use 5.15 kernel headers, and stable
toolchains now use 4.19 kernel headers
- Fortran support has been disabled on Microblaze, as the libgfortran
build at -O2 causes an internal compiler error.
All runtime tests are passing, except the ones for the new RISC-V
32-bit musl toolchain, for which Busybox fails to build due to an
interaction between musl-specific code in Busybox and musl. This issue
has been reported:
https://www.openwall.com/lists/musl/2024/03/03/2
The runtime tests are nevertheless included, with the hope that this
issue will reasonably quickly be resolved.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
The trace-cmd runtime test has a typo and fails with output:
Traceback (most recent call last):
File "/build/buildroot/support/testing/tests/package/test_trace_cmd.py", line 53, in test_run
self.assertEquals(exit_code, 0)
^^^^^^^^^^^^^^^^^
AttributeError: 'TestTraceCmd' object has no attribute 'assertEquals'. Did you mean: 'assertEqual'?
The issue can be reproduced with the command:
support/testing/run-tests \
-d dl -o output_test \
tests.package.test_trace_cmd
This commit fixes the issue by removing the extra 's'.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The hash files do not use trailing backslash \ to continue lines, so
we don't want them to be interpreted thusly, so we use 'read -r'
(SC2162).
The h_file is used twice in the same loop, once for reading from it,
and once just to print it, so there is no conflict (SC2094).
Integrer variables need not be quoted (SC2086). In any case, should
there be an actual issue and they be set empty, that would cause a
runtime issue, wether they be quoted or not.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
the user tables do not use trailing backslash \ to continue lines,
so we don't want them to be interpreted thusly, so we use 'read -r'
(SC2162).
Integer variables need not be quoted (SC2086). In any case, should
there be an actual issue and they be set empty, that would cause a
runtime issue, wether they be quoted or not.
The binary -o and -a ar perfectly defined in bash's test (SC2166).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
