Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).
Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.
In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.
Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.
Apply these checks for shell scripts:
- shellcheck;
- trailing space;
- consecutive empty lines;
- empty last line on file;
- newline at end of file.
Update the list of ignored warnings.
Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
On host without jq installed, test-pkg's output is as following:
$ ./utils/test-pkg -p mmc-utils
bootlin-armv5-uclibc [1/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
JQ IS
OK
bootlin-armv7-glibc [2/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
JQ IS
OK
bootlin-armv7m-uclibc [3/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
...
Since test-pkg handles this case we can hide this error message and have
proper output:
$ ./utils/test-pkg -p mmc-utils
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: OK
Signed-off-by: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Start counting the line numbers in 1 instead of 0, in case an error
must be printed.
Both the error about a developer entry with no file entry and the error
about a file entry with no developer entry actually belong to the
non-empty line previous the one being analysed, so in these cases print
the line number from the line before.
Also count empty and comment lines, so a developer fixing the file can
jump to the correct line (or the nearest one).
At same time standardize the messages, printing the line number
also in the case of a warning for a file that is not in the tree
anymore.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently 4 types of parsing errors/warnings can be found:
- entry for a file that is not in the tree anymore (warning)
- developer entry with no file entry (error)
- file entry with no developer (error)
- entry that is not a developer, a file or a comment (hard error)
Currently only the last one ends the script with -v with error code.
Make all 3 error types into hard errors and bail out at the first error
found, because the rest of the state machine is not designed to handle
malformed input.
Suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When a developer fixes an ignored warning from check-package, he/she
needs to update .checkpackageignore
By running './utils/docker-run make check-package' the developer
receives a warning about this.
Make that change easier to make, by adding a helper target on Makefile.
Add an option --failed-only to check-package that generates output in
the format:
<filename> <check_function> [<check_function> ...]
This is the very same format used by check-package ignore file.
Add the phony target .checkpackageignore
So one can update the ignore file using:
$ ./utils/docker-run make .checkpackageignore
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When a new check_function is added to check-package, often there are
files in the tree that would generate warnings.
An example is the Sob check_function for patch files:
| $ ./utils/check-package --i Sob $(git ls-files) >/dev/null
| 369301 lines processed
| 46 warnings generated
Currently these warnings are listed when calling check-package directly,
and also at the output of pkg-stats, but the check_function does not run
on 'make check-package' (that is used to catch regressions on GitLab CI
'check-package' job) until all warnings in the tree are fixed.
This (theoretically) allows new .patch files be added without SoB,
without the GitLab CI catching it.
So add a way to check-package itself ignore current warnings, while
still catching new files that do not follow that new check_function.
Add a file named .checkpackageignore to the buildroot topdir.
It contains the list of check_functions that are expected to fail for
each given intree file tested by check-package.
Each entries is in the format:
<filename> <check_function> [<check_function> ...]
These are 2 examples of possible entries:
package/initscripts/init.d/rcK ConsecutiveEmptyLines EmptyLastLine Shellcheck
utils/test-pkg Shellcheck
Keeping such a list allows us to have fine-grained control over which
warning to ignore.
In order to avoid this list to grow indefinitely, containing entries for
files that are already fixed, make each entry an 'expected to fail'
instead of just an 'ignore', and generate a warning if a check_function
that was expect to fail for a given files does not generate that
warning.
Unfortunately one case that do not generate warning is an entry for a
file that is deleted in a later commit.
By default, all checks are applied. The --ignore-list option allows to
specify a file that contains the list of warnings that should be
ignored.
The paths in the ignore file must be relative to the location of the
ignore file itself, which means:
- in the main Buildroot tree, the paths in the ignore file are
relative to the root of the main Buildroot tree
- in a BR2_EXTERNAL tree, if the ignore file is at the root of the
BR2_EXTERNAL, the paths it contains must be relative to that root
of the BR2_EXTERNAL
This is one more step towards standardizing the use of just 'make
check-package' before submitting patches to the list.
Cc: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .
The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).
Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.
At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
(for legacy handling) but is referenced in the package without a
comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
but it is not actually referenced.
There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.
Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'
At same time the script is created:
- add unit tests for it, they can be run using:
utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
using the script) and check-check-symbols (to check the script against
its unit tests)
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since genrandconfig no longer appears to support python2 we can
migrate the subprocess calls to use asyncio variants.
This has the advantage of allowing for runners like autobuild-run to
integrate directly into genrandconfig by calling the asyncio
gen_config using importlib instead of having to run genrandconfig as
a subprocess.
Using asyncio is advantageous here as it eliminates the requirement
for the runner to deal with blocking subprocess calls(by having to
use threading for example).
Also cleanup some unused functions/python2 compatibility shims.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
There are two legitimate cases to prefer ifdef over ifeq in package
recipes: command-line overrides are allowed for busybox and uclibc
configs.
Except for that, all package in tree already use ifeq, so warn the
developer adding/changing a package to use ifeq instead of ifdef, in
order to keep consistence across packages.
file.mk:2: use ifeq ($(SYMBOL),y) instead of ifdef SYMBOL
file.mk:5: use ifneq ($(SYMBOL),y) instead of ifndef SYMBOL
The difference between ifeq and ifdef is that ifdef doesn't expand
recursively.
Add comments to busybox and uclibc packages to avoid a warning in such
special cases.
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Warn the developer in the case the same config is declared more than
once in the same Config.in file.
But take into account the conditional code that lets the config be
visible and warn only when it is declared more than once in the same
conditions.
For instance, do not warn for:
if BR2_PACKAGE_BUSYBOX
config BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
endif
if !BR2_PACKAGE_BUSYBOX # kconfig doesn't support else
config BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
endif
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As the comment in package/perl/perl.mk instructs, bumping the perl
version must be propagated to utils/scancpan as well.
However, commit 7c1ef8129f (package/perl: bump to version 5.34.0)
forgot to do so.
Fixes: 7c1ef8129f
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
igh-ethercat comes with a small number of patched Linux kernel network
drivers, which aim at replacing the ones available in upstream Linux
kernel. All those drivers are provided only for specific kernel
releases. For example:
r8169-2.6.24-ethercat.c
r8169-2.6.24-orig.c
r8169-2.6.27-ethercat.c
r8169-2.6.27-orig.c
r8169-2.6.28-ethercat.c
r8169-2.6.28-orig.c
r8169-2.6.29-ethercat.c
r8169-2.6.29-orig.c
r8169-2.6.31-ethercat.c
r8169-2.6.31-orig.c
r8169-2.6.32-ethercat.c
r8169-2.6.32-orig.c
r8169-2.6.33-ethercat.c
r8169-2.6.33-orig.c
r8169-2.6.35-ethercat.c
r8169-2.6.35-orig.c
r8169-2.6.36-ethercat.c
r8169-2.6.36-orig.c
r8169-2.6.37-ethercat.c
r8169-2.6.37-orig.c
r8169-3.10-ethercat.c
r8169-3.10-orig.c
r8169-3.12-ethercat.c
r8169-3.12-orig.c
r8169-3.14-ethercat.c
r8169-3.14-orig.c
r8169-3.16-ethercat.c
r8169-3.16-orig.c
r8169-3.2-ethercat.c
r8169-3.2-orig.c
r8169-3.4-ethercat.c
r8169-3.4-orig.c
r8169-3.6-ethercat.c
r8169-3.6-orig.c
r8169-3.8-ethercat.c
r8169-3.8-orig.c
r8169-4.4-ethercat.c
r8169-4.4-orig.c
Obviously, this doesn't play well with the random configuration
testing done by utils/genrandconfig. This commit avoids this issue by
making sure we never build any of those drivers as part of the
genrandconfig generated configurations.
Fixes:
http://autobuild.buildroot.net/results/07b7475d780c067d99ee5618a5fd2bb024a5b4e7/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When BR2_XTENSA_CUSTOM=y is used with the internal toolchain, an
overlay file is mandatory, which genrandconfig can't provide. So we
simply disallow such configurations.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
a10disp will raise the following build failure with a mainline kernel:
cp: cannot stat '/home/autobuild/autobuild/instance-0/output-1/build/linux-5.17.15/include/video/sunxi_disp_ioctl.h': No such file or directory
Fixes:
- http://autobuild.buildroot.org/results/1f2607d6adece4d5dfe17fbdb032a2d228fc030f
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the internal toolchain backend, we have a choice..endchoice block
to allow the user to select the C library, between glibc, uClibc and
musl.
However, there are situations were no C library at all is
supported. In this case, the choice does not appear, and does not
allow to see the Config.in comments that are within the
choice..endchoice block and that may explain why no C library is
available.
For example, on RISC-V 32-bit, the only C library supported is glibc,
and the minimum kernel header version required by glibc on this
architecture is 5.4.0. In a future commit, we are going to add this
dependency on glibc (to fix build issues on configurations that have
headers < 5.4.0). But since glibc is the only supported C library on
RISC-V 32-bit, it means that the choice..endchoice for the C library
contains no entry, preventing from seeing the Config.in comment.
To address this issue, this commit adds a "dummy"
BR2_TOOLCHAIN_BUILDROOT_NONE option that shows up in the
choice..endchoice only when no C library is available. Thanks to this,
the choice..endchoice is never empty, and the Config.in comments can
be seen.
If the user keeps BR2_TOOLCHAIN_BUILDROOT_NONE selected, then the
build will anyway abort early because package/Makefile.in has a check
to verify that a C library is selected, and aborts the build if not.
Some could say that the problem should be resolved by instead
preventing the selection of headers < 5.4.0 on RISC-V 32-bit, but that
is difficult to do as the user can choose a custom header version, or
simply specific that (s)he wants to use the headers of the kernel
being built. In those situations, it's difficult to prevent selecting
headers < 5.4.0.
Prevent random configurations from triggering a build failure in our
autobuilders, by excluding that symbol from accepted configuration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: update genrandconfig]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit af494d92d3 (utils/genrandconfig: disable libopenssl without
atomics) intorduced a code-style issue that flake8 does not like:
$ make check-flake8
utils/genrandconfig:253:8: E713 test for membership should be 'not in'
1 E713 test for membership should be 'not in'
Fixes: af494d92d3https://gitlab.com/buildroot.org/buildroot/-/jobs/3045260108
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
libopenssl needs atomic or the build will fail (e.g. on sparcv8 without
libatomic):
${LDCMD:-/nvmedata/autobuild/instance-7/output-1/host/bin/sparc-buildroot-linux-uclibc-gcc} -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O0 -g2 -g2 -L. \
-o apps/openssl apps/asn1pars.o apps/ca.o apps/ciphers.o apps/cms.o apps/crl.o apps/crl2p7.o apps/dgst.o apps/dhparam.o apps/dsa.o apps/dsaparam.o apps/ec.o apps/ecparam.o apps/enc.o apps/engine.o apps/errstr.o apps/gendsa.o apps/genpkey.o apps/genrsa.o apps/nseq.o apps/ocsp.o apps/openssl.o apps/passwd.o apps/pkcs12.o apps/pkcs7.o apps/pkcs8.o apps/pkey.o apps/pkeyparam.o apps/pkeyutl.o apps/prime.o apps/rand.o apps/rehash.o apps/req.o apps/rsa.o apps/rsautl.o apps/s_client.o apps/s_server.o apps/s_time.o apps/sess_id.o apps/smime.o apps/speed.o apps/spkac.o apps/srp.o apps/storeutl.o apps/ts.o apps/verify.o apps/version.o apps/x509.o \
apps/libapps.a -lssl -lcrypto -ldl
/nvmedata/autobuild/instance-7/output-1/host/lib/gcc/sparc-buildroot-linux-uclibc/10.3.0/../../../../sparc-buildroot-linux-uclibc/bin/ld: ./libssl.so: undefined reference to `__atomic_fetch_sub_4'
It should be noted that openssl3 has added OPENSSL_DEV_NO_ATOMICS but
"this is intended for internal development only, to check the
refcounting is properly coded. It should never become a configuration
option, hence the name of the macro.":
503d4745a1
Atomics are not available in Buildroot if:
- architecture is 32 bit and something other than ARM or xtensa, and
- GCC < 4.8 or no threads or FLAT.
The nothreads case can theoretically happen in many different
situations, but in practice nobody disables threads. So the only
interesting case is the FLAT case. Since ARM and RISC-V 64 both have
atomics intrinsics, that leaves just m68k NOMMU as FLAT. So this is
truly a corner case.
The proper solution would be to patch GCC to also provide libatomic in
those cases.
- For nothreads, atomics are in fact not needed, so libatomic can simply
be implemented as stubs.
- For FLAT, it's probably just a matter of having a match to uclinux in
libatomic/configure.tgt.
Again, though, this happens only in such niche cases that it's not worth
working on it.
Fixes:
- http://autobuild.buildroot.org/results/bce526d538f43a541fdfbc0c9b4a7cecebbbc539
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Avoid the "Where did it put my builds?" step.
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
From flake8:
utils/genrandconfig:429:21: E703 statement ends with a semicolon
1 E703 statement ends with a semicolon
Fixes: d3e029575c
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Add a custom case to make sure that a random configuration with an empty
version for aufs-util doesn't fail.
Fixes:
- http://autobuild.buildroot.org/results/e242cf66a02983bcf6e95b37f8e458bd18aee683
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Improve commit 541e794a95 by adding a
custom case to make sure that a random configuration with an empty
platform for arm-trusted-firmware doesn't fail:
make_helpers/plat_helpers.mk:15: *** "Error: Unknown platform. Please use PLAT=<platform name> to specify the platform". Stop.
Fixes:
- http://autobuild.buildroot.org/results/1b67220008223d1bcbe70b76d643f9d04362ba6b
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In case of an unexpected error, we currently only print the exception as
an str(). For example, the recent issue with the glibc version check
only reported:
TypeError: cannot use a string pattern on a bytes-like object
That does not help in fixing the issue; the exception text is also not
usually very user-friendly either anyway.
We change the reporting to print the traceback, which in the glibc
version check mentioned above, the error is reported as:
Traceback (most recent call last):
File "./utils/genrandconfig", line 740, in <module>
ret = gen_config(args)
File "./utils/genrandconfig", line 676, in gen_config
if not is_toolchain_usable(configfile, toolchainconfig):
File "./utils/genrandconfig", line 186, in is_toolchain_usable
if StrictVersion('2.14') > StrictVersion(glibc_version):
File "/usr/lib/python3.8/distutils/version.py", line 40, in __init__
self.parse(vstring)
File "/usr/lib/python3.8/distutils/version.py", line 135, in parse
match = self.version_re.match(vstring)
TypeError: cannot use a string pattern on a bytes-like object
With this, the error is much easier to pinpoint (it's the last one that
is not in a system module).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Unless explicitly told otherwise, subprocess.check_output() returns
bytes objects [0].
When we try to check the C library version (to check the Linaro
toolchain is usable), genrandconfig currently fails with:
TypeError: cannot use a string pattern on a bytes-like object
So, as suggested in the python documentation, decocde() the output of
subprocess.check_output() before we can use it.
[0] https://docs.python.org/3/library/subprocess.html#subprocess.check_output
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We extend the modules search path to be able to load the package
metadata. Currently, it is only restored when loading those
succeeded, not when it failed.
Restore it to its previous state also in case of error, to avoid
leaking the path further.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add custom cases to make sure that a random configuration with an empty
git, tarball location or version for xenomai doesn't fail. It reverts to
BR2_PACKAGE_XENOMAI_LATEST_VERSION in these cases.
Fixes:
- http://autobuild.buildroot.org/results/19419759b4add0197b4e629d0b2216c2a07624b2
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a custom case to make sure that a random configuration with an empty
git location for refpolicy doesn't fail. It reverts to
BR2_PACKAGE_REFPOLICY_UPSTREAM_VERSION in that case.
Fixes:
- http://autobuild.buildroot.org/results/98da07a1f9b88e2e8a028a55c8cf76c6ebc28304
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fix typo on line 23 `get-developpers` to `get-developers`
Signed-off-by: Jesse Taube <Mr.Bossman075@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In preparation to the introduction of test cases for the
get-developers script, make it possible to pass a custom DEVELOPERS
file. A normal user of get-developers will most likely never use this,
but the test suite will use it.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: extracted from a larger patch from Ricardo, initially posted
at https://patchwork.ozlabs.org/project/buildroot/patch/20220528014832.289907-1-ricardo.martincoski@gmail.com/]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit "45aabcddc5 utils/get-developers: really make it callable from
elsewhere than the toplevel directory" had a unforeseen side-effect:
the fact that check-developers with no arguments used to validate the
syntax of the DEVELOPERS no longer works. Indeed, the
parse_developers() call was moved *after* the check that verifies if
at least one action is passed. Due to this, the check-DEVELOPERS
verification in Gitlab CI, which invokes get-developers without
argument to validate the DEVELOPERS file... no longer does anything
useful.
In order to fix this, we introduce an explicit action in
get-developers to request the validation of the DEVELOPERS file. The
implementation is trivial, as the validation is already done by
calling parse_developers(), but at least now we have an action that
does "nothing", except execute until the validation is done.
This is also much more explicit than "invoke get-developers without
arguments to just do validation".
The initial investigation of the issue was done by Ricardo Martincoski
<ricardo.martincoski@gmail.com> and reported at
https://patchwork.ozlabs.org/project/buildroot/patch/20220528014832.289907-1-ricardo.martincoski@gmail.com/,
but we have chosen to introduce a separate -v option rather than
making the validation part of the existing -c action, which serves a
different purpose.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a custom case to make sure that a random configuration with an
empty tarball location for OP-TEE OS doesn't fail. It reverts to
BR2_TARGET_OPTEE_OS_LATEST in that case.
Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The support for this architecture has been removed from the upstream
Linux kernel, as of commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aec499c75cf8e0b599be4d559e6922b613085f8f
Which states:
The nds32 architecture, also known as AndeStar V3, is a custom
32-bit RISC target designed by Andes Technologies. Support was added
to the kernel in 2016 as the replacement RISC-V based V5 processors
were already announced, and maintained by (current or former) Andes
employees.
As explained by Alan Kao, new customers are now all using RISC-V,
and all known nds32 users are already on longterm stable kernels
provided by Andes, with no development work going into mainline
support any more.
There has also been little to no maintenance done in Buildroot for
this architecture in recent times, so let's follow the Linux kernel
community decision and drop support for this CPU architecture.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
If we don't have a DTS name or path we need to also disable cuimage
to disable DTS support as BR2_LINUX_KERNEL_CUIMAGE requires DTS
support.
Fixes:
linux/linux.mk:591: *** No kernel device tree source specified, check your BR2_LINUX_KERNEL_INTREE_DTS_NAME / BR2_LINUX_KERNEL_CUSTOM_DTS_PATH settings. Stop.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
If the kconfig values being replaced are not empty we should not
try to fix them.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Internally genrandconfig will use gettimeofday when generating a
KCONFIG_SEED, since autobuild-run executes genrandconfig at the same
time for multiple autobuilder runners this could potentially result
in the same KCONFIG_SEED being generated for those test runs started
at the same time.
To ensure this doesn't happen set the KCONFIG_SEED using the urandom
entropy source.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This should have been BR2_TARGET_ROOTFS_EXT2 as the
BR2_TARGET_ROOTFS_EXT2_GEN variable is an integer variable used
to indicate the ext2/ext3/ext4 variant.
Fixes:
- http://autobuild.buildroot.net/results/5d7/5d7833212bd8a2b575945f848d8c91dabe7d3c30
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Until commit "arch/Config.in.sh: fixup MMU selection" in this series,
SH2A could either be used with BR2_USE_MMU disabled or BR2_USE_MMU
enabled.
The later made absolutely no sense, since SH2A does not have a MMU:
MMU support was introduced starting from SH3 according to
https://en.wikipedia.org/wiki/SuperH#SH-3
Also, since commit 22d5501e03 ("arch:
tidy up binary formats config"), which was merged in Buildroot
2015.05, the architecture tuple used when BR2_sh2a=y and BR2_USE_MMU
disabled is sh2a-buildroot-uclinux-uclibc, and this was already
unsupported back in the days of Buildroot 2015.08 and binutils 2.24,
causing the build to fail with:
*** BFD does not support target sh2a-buildroot-uclinux-uclibc.
just like it fails to build today with recent version of binutils.
So, this has been broken since 2015.08, and nobody complained. SH2A is
seldom used, so it's time to kill it.
It is worth mentioning that there had been an attempt at resurrecting
SH2 support around 2015 (see https://lwn.net/Articles/647636/) as part
of the J2 core. This effort led to the addition of FDPIC support for
SH2A in the musl C library (and therefore proper ELF binaries, with
shared libraries), but that was never supported in Buildroot. Now that
the J2 project is essentially dead, there is no reason to bother with
this.
Fixes:
http://autobuild.buildroot.net/results/63d01d33ae30f86b63b9f42a9fea116f2f3e9005/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently we only test a limited set of toolchains that are mostly
prebuilt, add a flag to allow using randconfig for randomizing
additional toolchain settings instead of randpackageconfig.
To avoid invalid configs we need to add additional config validation
filtering and fixups. Although the loop around 'make randconfig'
attempts does make sure we eventually end up with a valid configuration,
there is a bias towards e.g. disabling the kernel. It would be possible
to e.g. always force BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG=y, but
that removes the possibility for a toolchains-csv config to add a valid
custom defconfig. (Note that currently this isn't possible anyway since
'make randpackageconfig' is used if a toolchains-csv is provided, but
eventually we want to use 'make randconfig' also if a toolchains-csv is
provided.)
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Some packages only have a home page properly set inside project_urls.
Squelch flake8's E127, because a visual indent here is really nicer.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[yann.morin.1998@free.fr: simplify getting home_page fallbacks]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
As already done for {FOO}_DEPENDENCIES in commit
4910a175b3, check that {FOO}_CONF_OPTS are
never overridden in a conditional
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Since this is a helper script there is not much reason to show the
command that's been issued. Furthermore, the incantation has been
slightly extended since the script was introduced.
The only interesting reason to print the command is to know what image
it is being spawned into. However, this is prominently displayed by
docker the first time the script is run, as it can't find the image
locally and has to fetch it first. Afterwards, users can still use
'docker image ls' to see what images they have locally.
So let's remove 'set -x' before running docker.
Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
[yann.morin.1998@free.fr: reword and expand commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the docker-run script starts a container based on the
reference image, in which the user has UID:GID set to 1000:1000,
which may or may not be the same as local user uses, which may
prevent the commands in the container from creating files, which
foils the plan of using the container to run a build...
Additionally, the paths in the container differ from the paths
the user expects, as the current Buildroot tree is mounted over
the in-container user home directory. This is a bit confusing...
Finally, the container is left hanging around after the command
finishes, and thus there are as many lingering containers as the
user runs docker-run. This is not very nice...
We fix all of that (yeah, should be different patches, but meh):
- we use --mount instead of -v, which allows us to bind-mount
the Buildroot tree at the same place in the container, as
Docker will create the destination if it does not exist, while
-v seems to expect it to exist [0].
- as a consequence, we can also set the working directory as the
Buildroot top-directory;
- use --user to force the same UID:GID in the container as the
local user, so that files created in the container belong to
the local user, and so that files from the local user are
accessible from the container;
- use --rm to remove the container once it terminates; starting
a new container is very quick anyway, so it is as good as
re-using a previous container.
[0] the documentation is not clear about that. It clearly states
that the host directory (i.e. the origin, the source) is created
if missing, but it says nothing of the destination:
https://docs.docker.com/engine/reference/commandline/run/#mount-volume--v---read-only
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Giulio Benetti <giulio.benetti@benettiengineering.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Tested-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Without -i, the container cannot be used interactively. Allow using it as
in:
$ ./utils/docker-run
+ exec docker run -v /home/ceresoli/devel/buildroot:/home/br-user -it registry.gitlab.com/buildroot.org/buildroot/base:20220206.1756
br-user@1e9508a4ccbb:~$ ls
arch board boot CHANGES Config.in Config.in.legacy configs COPYING DEVELOPERS docs fs linux Makefile Makefile.legacy output package README support system toolchain utils
br-user@1e9508a4ccbb:~$
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
/usr/bin/bash does not exist on Ubuntu 2018.04. Use the more portable
alternative "#!/usr/bin/env bash" which we are already using in quite
a few scripts.
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Fixes: 242e9d72e7 ("utils/docker-run: new script")
[yann.morin.1998@free.fr: use "#!/usr/bin/env bash", not "@!/bin/bash"]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The option was removed by commit 306839586b (package/python: drop target
package).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
While committing the shellcheck feature, it was changed to output the
full shellcheck output even at verbosity level 1. However, the
expectation of the shellcheck test was not updated accordingly.
Do that now, simply merging all the shellcheck output in a single
string.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
For simplicity, when shellcheck returns one or more warnings, count it
as a single check-package warning.
The developer can get the full output either by running shellcheck
directly or by calling check-package with -v.
Examples:
|$ ./utils/docker-run utils/check-package --include Shellcheck package/polkit/S50polkit
|package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings
|51 lines processed
|1 warnings generated
|$ ./utils/docker-run utils/check-package --include Shellcheck -v package/polkit/S50polkit
|package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings
|In package/polkit/S50polkit line 43:
|< tab >start|stop|restart|reload)
| ^----^ SC2221: This pattern always overrides a later one on line 45.
|In package/polkit/S50polkit line 45:
|< tab >reload)
| ^----^ SC2222: This pattern never matches because of a previous pattern on line 43.
|For more information:
| https://www.shellcheck.net/wiki/SC2221 -- This pattern always overrides a l...
| https://www.shellcheck.net/wiki/SC2222 -- This pattern never matches becaus...
|51 lines processed
|1 warnings generated
NOTICE: shellcheck results depends on the version of the tool.
This is why the examples above run inside the docker image.
Also update .gitlab-ci.yml with the docker image after the change of
the previous commit. We don't actually use shellcheck in CI, but the
image from .gitlab-ci.yml is used by the docker-run script and we use
that to run shellcheck.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: make sure a single -v is enough to get shellcheck output;
update docker image tag in .gitlab-ci.yml]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Enable the common checks:
- consecutive empty lines
- empty last line
- missing new line at end of file
- trailing space
- warn for executable files, with the hint to instead use
'$(INSTALL) -D -m 0755' in the .mk file
Check indent with tabs:
- add a simple check function to warn only when the indent is done
using spaces or a mix of tabs and spaces. It does not check indenting
levels, but it already makes the review easier, since it
diferentiates spaces and tabs.
Check variables:
- check DAEMON is defined
- when DAEMON is defined, check the filename is in the form S01daemon
- when PIDFILE is defined, expect it to be in /var/run and defined
using $DAEMON.
Also add unit test for this.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: avoid 'del NotExecutable_base' by importing the module instead
of the class; refer to manual in warnings]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently there are no .mk, Config.in, .patch or .hash files with
executable permissions in the tree.
But we don't want to have that.
So warn when a file checked by check-package has executable permission.
This check will be reused when testing SysV init scripts in the tree.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: use context manager for temp dir so it gets deleted]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Some file formats have well-established syntax checkers.
One example of this is the tool 'shellcheck' that can analyse shell
scripts for common mistakes.
There is no reason to reimplement such tools in check-package, when we
can just call them.
Add the ability to check-package to call external tools that will run
once for each file to be analysed.
For simplicity, when the tool generated one or more warnings, count it
as a single warning from check-package, that can display something like
this:
|$ ./utils/check-package package/unscd/S46unscd
|package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings
|25 lines processed
|1 warnings generated
|$ ./utils/check-package -vvvvvvvvvvvvvvvv package/unscd/S46unscd
|package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings
|In package/unscd/S46unscd line 9:
| printf "Starting ${NAME}: "
| ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
|In package/unscd/S46unscd line 11:
| [ $? -eq 0 ] && echo "OK" || echo "FAIL"
| ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
|In package/unscd/S46unscd line 14:
| printf "Stopping ${NAME}: "
| ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
|In package/unscd/S46unscd line 16:
| [ $? -eq 0 ] && echo "OK" || echo "FAIL"
| ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
|For more information:
| https://www.shellcheck.net/wiki/SC2059 -- Don't use variables in the printf...
| https://www.shellcheck.net/wiki/SC2181 -- Check exit code directly with e.g...
|25 lines processed
|1 warnings generated
In this first commit, add only the ability for check-package to call
external tools and not an example of such tool, as adding each tool to
call may need update to the docker image and can lead to it's own
discussion on how to implement.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a small script to run commands in the same docker image used in the
GitLab CI.
For instance, one can run check-package unit tests without installing
pytest directly in the host:
$ ./utils/docker-run python3 -m pytest -v utils/checkpackagelib/
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: fix shellcheck errors; add exec]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
So anyone willing to contribute to check-package can run all tests in
less than 1 second by using:
$ python3 -m pytest -v utils/checkpackagelib/
Most test cases are in the form:
@pytest.mark.parametrize('testname,filename,string,expected', function)
- testname: a short description of the scenario tested, added in order
to improve readability of the log when some tests fail
- filename: the filename the check-package function being tested thinks
it is testing
- string: the content of the file being sent to the function under test
- expected: all expected warnings that a given function from
check-package should generate for a given file named filename and
with string as its content.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
size-stats-compare gives an overview of the size increase/decrease between
two cases, based on packages-file-list.txt. The 'detail' mode gives info per
file, otherwise per package.
But sometimes, you want the detailed per-file info, but only for a specific
package. Since the detailed output no longer lists the package name, you
cannot simply grep for it. A workaround was to filter the input
packages-file-list.txt's first, and then pass these filtered versions to
size-stats-compare.
Make this easier by adding the package name next to the filename in detailed
output. This allows grep'ing normally.
For example:
$ utils/size-stats-compare orig new -t 100 -d | grep ebtables
-67712 removed ebtables lib/ebtables/libebtc.so
-66764 removed ebtables lib/ebtables/libebt_nat.so
-66752 removed ebtables sbin/ebtables
-66704 removed ebtables lib/ebtables/libebt_arp.so
-66700 removed ebtables lib/ebtables/libebt_stp.so
-66700 removed ebtables lib/ebtables/libebt_among.so
-66684 removed ebtables lib/ebtables/libebt_ip.so
-66676 removed ebtables lib/ebtables/libebt_limit.so
-66656 removed ebtables lib/ebtables/libebt_log.so
-66648 removed ebtables lib/ebtables/libebt_mark.so
-66636 removed ebtables lib/ebtables/libebt_pkttype.so
-66604 removed ebtables lib/ebtables/libebt_vlan.so
-66588 removed ebtables lib/ebtables/libebt_ulog.so
-66588 removed ebtables lib/ebtables/libebt_nflog.so
-66584 removed ebtables lib/ebtables/libebt_arpreply.so
-66544 removed ebtables lib/ebtables/libebt_ip6.so
-66540 removed ebtables lib/ebtables/libebt_802_3.so
-66536 removed ebtables lib/ebtables/libebt_standard.so
-66524 removed ebtables lib/ebtables/libebt_mark_m.so
-66524 removed ebtables lib/ebtables/libebt_redirect.so
-66452 removed ebtables lib/ebtables/libebtable_filter.so
-66452 removed ebtables lib/ebtables/libebtable_broute.so
-66452 removed ebtables lib/ebtables/libebtable_nat.so
45 ebtables etc/ethertypes
66752 added ebtables usr/sbin/ebtablesd
66752 added ebtables usr/sbin/ebtables-legacy
66752 added ebtables usr/sbin/ebtablesu
200840 added ebtables usr/lib/libebtc.so.0.0.0
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
print_result is juggling with entry[x][y] which is not very readable.
While a better solution would be to use a class and reference named
attributes, that would require some bigger changes in the script.
Instead, make a minimal improvement by assigning the entry[x][y] values to
intermediate variables. Store them in a dict for easy usage from a format
string.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Commit 1ba85b7f87 (support/download: add explicit no-hash support)
introduced the 'none' hash type, in an attempt to make hash files
mandatory, but not failing on archives localy generated, like those
for git or svn repositories, especially for those packages where a
version choice was present, which would allow for either remote
archives for which we'd have a hash or VCS trees for which we could
not have a hash for the localy generated archive.
Indeed, back in the time, we did not have a mean to generate
reproducible archives, so having a hash file without a hash for
thosel ocally generated archives would trigger an error in the
hash-checking machinery.
But now, low-and-behold, we do know how to generate those archives,
and we have a mechanism to explicitly exclude some archives from being
hash-checked (e.g. when the version string itself can be user-provided).
As such, the 'none' hash type no longer has any raison d'être, we do not
use it in-tree, and its use in a br2-external tree is most probably
inexistent (as is the use of hash files alotgether most probably).
So we simply drop the support for that.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Thomas: drop support in checkpackagelib, as reported by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
JSON parser are usually rather strict when parsing their input. Both
jq and the python json module choke on non-conformant inputs.
Commit e4c284e6b9 (package/pkg-utils: escape \ in generated legal-info)
fixed the \-escaping case, but we want to ensure that we do not have
other data that would be improperly encoded.
To more easily catch issues, also run show-info and send its output
through jq, to try and validate the output.
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The seperation of the fields in the hash file should be 2 spaces for
consistency.
Since a large number of hash files still violate this rule, exclude it
from "make check-package" (and thus from CI).
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout:
- Move it to a separate class, so it can be excluded.
- Exclude it from "make check-package"
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Python 2 is EOL sice 2020 [1], it's still available on distros, but may
not be installed by default (as being replaced by python3). A
compatibility symlink python -> python3 may not be installed either.
Convert the shebang line to the usual /usr/bin/env based line.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, by following the instructions in the manual and querying for
developers for a patch that changes path
package/foobar
the script reports both developers that have these entries in the
DEVELOPERS file:
F: package/foo/
F: package/foobar/
Starting from commit "afc112b0e4 utils/getdeveloperlib.py: fix issue
with hasfile()" get-developers script uses os.path.abspath() and
os.path.relpath().
The catch is that those functions return the absolute path and the
relative path without the trailing slash.
When the paths associated to a developer are then compared to the paths
a patch touches, using the string.startswith(), any substring returns
True, leading to developers for package/foo/ being wrongly reported
for package/foobar/ .
Fix this by re-adding the trailing slash after using relpath().
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: James Knight <james.d.knight@live.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently this .mk snippet results in unexpected behavior from
check-package:
|VAR_1 = VALUE1
|ifeq (condition)
|VAR_1 := $(VAR_1), VALUE2
|endif
Fix commit "163f160a8e utils/{check-package, checkpackagelib}:
consistently use raw strings for re.compile" that ended up doing this:
- CONCATENATING = re.compile("^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")
+ CONCATENATING = re.compile(r"^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")
But raw strings do not expect escaping when referencing \1 and the
pattern ends up searching for a raw '\\1' instead of an occurrence of
the first pattern inside parenthesis.
|$ python3
|Python 3.8.10 (default, Sep 28 2021, 16:10:42)
|[GCC 9.3.0] on linux
|Type "help", "copyright", "credits" or "license" for more information.
|>>> import re
|>>> p1 = re.compile('(foo)bar\\1')
|>>> p2 = re.compile(r'(foo)bar\\1')
|>>> p3 = re.compile(r'(foo)bar\1')
|>>> s1 = 'foobarfoo'
|>>> s2 = 'foobar\\1'
|>>> print(p1.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p2.search(s1))
|None
|>>> print(p3.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p1.search(s2))
|None
|>>> print(p2.search(s2))
|<re.Match object; span=(0, 8), match='foobar\\1'>
|>>> print(p3.search(s2))
|None
|>>>
So use '\1' instead of '\\1' in the raw string.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Check that {FOO}_DEPENDENCIES are never overriden in a conditional
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Check typo in define to detect SMAKE_LINUX_CONFIG_FIXUPS in smack
(fixed by 41e2132fbe)
The new expression will catch "SMAKE_CONF_OPTS" as well as
"define SMAKE_LINUX_CONFIG_FIXUPS"
Two modifications were made:
- add (define\s+)? which will match "define " but also an empty value.
Thanks to this, the second group will always contain the variable
name.
- remove \s*(\+|)= which seems superfluous
Also, add GCC_TARGET in ALLOWED variable to avoid the following
warnings:
arch/arch.mk:12: possible typo: GCC_TARGET_ARCH -> *ARCH*
arch/arch.mk:13: possible typo: GCC_TARGET_ABI -> *ARCH*
arch/arch.mk:14: possible typo: GCC_TARGET_NAN -> *ARCH*
arch/arch.mk:15: possible typo: GCC_TARGET_FP32_MODE -> *ARCH*
arch/arch.mk:16: possible typo: GCC_TARGET_CPU -> *ARCH*
arch/arch.mk:17: possible typo: GCC_TARGET_FPU -> *ARCH*
arch/arch.mk:18: possible typo: GCC_TARGET_FLOAT_ABI -> *ARCH*
arch/arch.mk:19: possible typo: GCC_TARGET_MODE -> *ARCH*
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The message attribute does not exist in python3, see PEP-0352:
https://www.python.org/dev/peps/pep-0352/
Fixes:
Traceback (most recent call last):
File "utils/scanpypi", line 743, in <module>
main()
File "utils/scanpypi", line 693, in main
if 'buildutils' in err.message:
AttributeError: 'ImportError' object has no attribute 'message'
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
genrandconfig is used by the Buildroot autobuilders to generate
semi-random configurations that we build test. As part of this, we use
"make randpackageconfig" to randomize the selection of packages,
together with a KCONFIG_PROBABILITY value, which indicates the
probabibility for each option to be enabled. This probability is
itself randomized, between 1% and 30% for every build.
However, with our increasing number of packages (over 2900), when we
use a 30% probability for options to be enabled, it means a *lot* of
options are enabled, causing very large configurations to be
tested. These configurations are not very realistic, and they take
ages to build on our autobuilders: we have builds that take 4, 5 or
even 7 hours to build.
In order to test a larger number of configurations and therefore a
larger variety of configurations, this commit reduces the maximum
probability to 20%.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In commit
40bb37bd70 ("utils/getdeveloperlib.py:
use relative paths for files"), the Developer class was changed to use
relative paths, including for its .hasfile() method.
However the check_developers() function of getdeveloperlib.py was not
updated accordingly, and continued to pass absolute paths. This caused
"get-developers -c" to return the entire list of files in Buildroot as
being unmaintained, as none of them were matching the file listed in
the DEVELOPERS file.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In Python 3.x, check_output() returns a "bytes" array, and not a
string. Its result needs to be decoded to be turned into a
string. Without this fix, "get-developers -c" bails out with:
Traceback (most recent call last):
File "/home/thomas/projets/buildroot/./utils/get-developers", line 105, in <module>
__main__()
File "/home/thomas/projets/buildroot/./utils/get-developers", line 53, in __main__
files = getdeveloperlib.check_developers(devs)
File "/home/thomas/projets/buildroot/utils/getdeveloperlib.py", line 280, in check_developers
files = subprocess.check_output(cmd).strip().split("\n")
TypeError: a bytes-like object is required, not 'str'
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
ee8b680816 ("utils/scanpypi: use python3 explicitly") started to use python3,
thus compatibility can be removed:
from __future__ import print_function
from __future__ import absolute_import
Tested with python3 -m py_compile.
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Python 2 is EOL sice 2020 [1], it's still available on distros, but may not
be installed by default (as being replaced by python3).
Thus remove compatibility imports:
from __future__ import print_function
from __future__ import absolute_import
Tested with python3 -m py_compile.
[1] https://www.python.org/doc/sunset-python-2/
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Force relative file path resolution of DEVELOPERS file entries to use
forward-slash separators since pattern matching assumes forward slashes.
This is to help permit uses invoking `get-developers` on Platforms where
`os.sep` may not be a forward slash.
Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Explicitly indicate the file encoding to UTF-8 for the DEVELOPERS
document. This prevents Unicode decoding errors when printing E-Mail
entries with Unicode characters on systems using an alternative default
encoding (e.g. 'CP1252').
This corrects the following observed error:
$ ./utils/get-developers outgoing/*
Traceback (most recent call last):
File "utils\get-developers", line 105, in <module>
__main__()
File "utils\get-developers", line 47, in __main__
devs = getdeveloperlib.parse_developers()
File "...\buildroot\utils\getdeveloperlib.py", line 239, in parse_developers
for line in f:
File "...\Python<ver>\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6659: character maps to <undefined>
Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, running test-pkg is only done locally on the developers
machine.
In a follow up commit, we'll add the possibility to run test-pkg in a
gitlab-ci pipeline and, to speed things up, with one job per buildable
configuration.
As such, we will need to make sure that test-pkg only prepares the
configurations, and that it does not build them.
Add such a mode, with a new option, --prepare-only
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When the config fragment provided by the user is not usable with a
specific toolchain configuration, the resulting .config file was kept
around.
In a follow up commit, we'll need to know, from outside test-pkg, if a
specific configuration was indeed usable or not.
So, unless if the user actually requested to keep the build directories,
remove the .config file when it contains a configration that would be
skipped.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Commit e43c050944 introduced two flake8
errors:
utils/scanpypi:300:26: E231 missing whitespace after ','
utils/scanpypi:302:9: F841 local variable 'setup' is assigned to but never used
The first one is easily fixed. The second one needs a little bit of
explanation. Before commit e43c0509, the return value of
imp.load_module() was used to be able to explicitly call the 'setup'
function in it in case the metadata was not populated. Since that
commit, calling that function is no longer needed, since setup.py is
executed in exactly the same way as when it's run from the command line,
so if that doesn't work, it's completely broken anyway. Therefore, we
can simply discard the return value of imp.load_module().
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In case the setup.py file of a python package does not directly call the
'setup' method, utils/scanpypi was hoping there be a 'main' function which
would do the work, normally called via a construct like:
if __name__ == '__main__':
main()
However, this construct is nonstandard, and there are packages in PyPI which
call 'setup()' directly from the 'if' statement, without a main() method.
But scanpypi does not actually need to make such assumption: when loading
the module, it can decide the name to be '__main__', just as if setup.py
would be loaded interactively.
Additionally, remove some logic seemingly related to the previous trick of
calling 'main'. There should not be a problem in keeping already loaded
modules in sys.modules, as this is the purpose of sys.modules.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Even though the directory containing a package's setup.py was added to
sys.path, some setup.py implementations rely on the fact that it is placed
in sys.path[0].
An example package is 'cram' which failed to be added with scanpypi:
Traceback (most recent call last):
File "utils/scanpypi", line 756, in <module>
main()
File "utils/scanpypi", line 703, in main
package.load_setup()
File "utils/scanpypi", line 303, in load_setup
setup = imp.load_module('setup', s_file, s_path, s_desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 44, in <module>
long_description=long_description(),
File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 20, in long_description
return open(os.path.join(sys.path[0], 'README.rst')).read()
FileNotFoundError: [Errno 2] No such file or directory: '.../buildroot/utils/README.rst'
The corresponding code from cram's setup.py is:
def long_description():
"""Get the long description from the README"""
return open(os.path.join(sys.path[0], 'README.rst')).read()
Indeed, the Python documentation says:
https://docs.python.org/3.8/library/sys.html#sys.path
"...
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to invoke
the Python interpreter.
..."
Fix this by inserting explicitly at index 0 instead of appending to
sys.path.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some packages such as python-idna has a LICENSE.md file:
https://github.com/kjd/idna/blob/master/LICENSE.md
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
... similar to BR2_ENABLE_DEBUG.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Since 810ba387be, some form of these options are enable
by default. Specifically:
- Kept FORTIFY level 2 option as the default is now level 1.
- Removed all SSP options as the default now uses the best
option based on toolchain support.
- Similar to SSP, for RELRO, the default now uses the best
option based on toolchain support.
- Completely drop PIC PIE as it defaults =y
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
scanpypi is python3 compatible. In addition, it executes the setup.py
of Python modules to extract the relevant information. Since these are
more and more commonly using python3 constructs, using "python" to run
scanpypi causes problems on systems that have python2 installed as
python, when trying to parse setup.py scripts with python3 constructs.
Fixes part of #13516.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When a developer has package/pkg-<infra>.mk assigned to him/her in the
DEVELOPERS file, this has 3 implications:
(1) Patches adding new packages using this infrastructure are Cc'ed
to this developer. This is done by the analyze_patch() function,
which matches the regexp r"^\+\$\(eval
\$\((host-)?([^-]*)-package\)\)$" in the patch, i.e where an
added line contains a reference to the infra maintained by the
developer.
(2) Patches touching the package/pkg-<infra>.mk file itself are Cc'ed
to this developer.
(3) Any patch touching a package using this infra are also Cc'ed to
this developer.
Point (3) causes a significant amount of patches to be sent to
developers who have package/pkg-generic.mk and
package/pkg-autotools.mk assigned to them in the DEVELOPERS
file. Basically, all patches touching generic or autotools packages
get CC'ed to such developers, which causes a massive amount of patches
to be received.
So this patch adjusts the getdeveloperlib.py to drop point (3), but
preserves point (1) and (2). Indeed, it makes sense to be Cc'ed on new
package additions (to make a review that they use the package
infrastructure correctly), and it makes sense to be Cc'ed on patches
that touch the infrastructure code itself.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 40bb37bd70 refactored get-developers, and now the 'os' module is
no longer needed, but still imported:
utils/get-developers:6:1: F401 'os' imported but unused
1 F401 'os' imported but unused
Drop it now.
Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.
parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.
This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.
Before this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
"name": "stm32f469_disco",
"path": "configs/stm32f469_disco_defconfig",
"developers": []
}
After this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
"Floris Bos <bos@je-eigen-domein.nl>",
"Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
"name": "stm32f469_disco",
"path": "configs/stm32f469_disco_defconfig",
"developers": [
"Christophe Priouzeau <christophe.priouzeau@st.com>"
]
}
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Instead of open-coding Developers.hasfile() in utils/get-developers,
use it directly.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
An 'else' or 'elif' clause inside a make conditional should not be indented
in the same way as the if/endif clause. check-package did not recognize the
else statement and expected an indentation.
For example:
ifdef FOOBAR
interesting
else
more interesting
endif
would, according to check-package, need to become:
ifdef FOOBAR
interesting
else
more interesting
endif
Treat 'else' and 'elif' the same as if-like keywords in the Indent test, but
take into account that 'else' is also valid shell, so we need to correctly
handle line continuation to prevent complaining about the 'else' in:
ifdef FOOBAR
if true; \
... \
else \
... \
fi
endif
We don't add the 'else' and 'elif' statements to start_conditional, because
it would cause incorrect nesting counting in class OverriddenVariable.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
pkg-stats is not able anymore to set the developers for defconfigs and
packages. This issue is introduced with
ae86067a15. The hasfile() method from
Developer object tries to check an absolute path against a relative path.
Convert the filepath to be checked also into an absolute path.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Normally the kconfig stings would end up empty and cause a build
error. This patch provides test files to allow testing the creation
of uboot environment and script bin files from user provided txt files.
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
