Commit Graph

265 Commits

Author SHA1 Message Date
Fabrice Fontaine
485aa2d691 utils/genrandconfig: add mxs-bootlets custom handling
Add custom cases to make sure that a random configuration with an empty
git or tarball location for mxs-bootlets doesn't fail. It reverts to
BR2_TARGET_MXS_BOOTLETS_FREESCALE in that case.

Fixes:
 - http://autobuild.buildroot.org/results/dcbeb73d152c79c18b1fa3bef3b4fa07635d7b36
 - http://autobuild.buildroot.org/results/f61cf3fc58db9ded5ec42ebf4a9847584700698d

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-09-06 21:58:49 +02:00
Arnout Vandecappelle
1fc299ad6a utils/genrandconfig: remove spurious semicolon
From flake8:
utils/genrandconfig:429:21: E703 statement ends with a semicolon
1     E703 statement ends with a semicolon

Fixes: d3e029575c

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-08-24 13:21:39 +02:00
Fabrice Fontaine
fee46b54e7 utils/genrandconfig: add aufs-util handling
Add a custom case to make sure that a random configuration with an empty
version for aufs-util doesn't fail.

Fixes:
 - http://autobuild.buildroot.org/results/e242cf66a02983bcf6e95b37f8e458bd18aee683

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-24 13:16:27 +02:00
Fabrice Fontaine
d3e029575c utils/genrandconfig: improve ARM trusted firmware fixup
Improve commit 541e794a95 by adding a
custom case to make sure that a random configuration with an empty
platform for arm-trusted-firmware doesn't fail:

make_helpers/plat_helpers.mk:15: *** "Error: Unknown platform. Please use PLAT=<platform name> to specify the platform".  Stop.

Fixes:
 - http://autobuild.buildroot.org/results/1b67220008223d1bcbe70b76d643f9d04362ba6b

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 23:55:49 +02:00
Yann E. MORIN
b6bfa3f744 utils/genrandconfig: dump traceback for unhandled exceptions
In case of an unexpected error, we currently only print the exception as
an str(). For example, the recent issue with the glibc version check
only reported:
    TypeError: cannot use a string pattern on a bytes-like object

That does not help in fixing the issue; the exception text is also not
usually very user-friendly either anyway.

We change the reporting to print the traceback, which in the glibc
version check mentioned above, the error is reported as:

    Traceback (most recent call last):
      File "./utils/genrandconfig", line 740, in <module>
        ret = gen_config(args)
      File "./utils/genrandconfig", line 676, in gen_config
        if not is_toolchain_usable(configfile, toolchainconfig):
      File "./utils/genrandconfig", line 186, in is_toolchain_usable
        if StrictVersion('2.14') > StrictVersion(glibc_version):
      File "/usr/lib/python3.8/distutils/version.py", line 40, in __init__
        self.parse(vstring)
      File "/usr/lib/python3.8/distutils/version.py", line 135, in parse
        match = self.version_re.match(vstring)
    TypeError: cannot use a string pattern on a bytes-like object

With this, the error is much easier to pinpoint (it's the last one that
is not in a system module).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 22:19:52 +02:00
Yann E. MORIN
12e4f7c5c4 utils/genrandconfig: fix checking host glibc version
Unless explicitly told otherwise, subprocess.check_output() returns
bytes objects [0].

When we try to check the C library version (to check the Linaro
toolchain is usable), genrandconfig currently fails with:
    TypeError: cannot use a string pattern on a bytes-like object

So, as suggested in the python documentation, decocde() the output of
subprocess.check_output() before we can use it.

[0] https://docs.python.org/3/library/subprocess.html#subprocess.check_output

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 22:19:45 +02:00
Fabrice Fontaine
2d3d130bc1 utils/genrandconfig: add opensbi custom handling
Add custom cases to make sure that a random configuration with an empty
git, tarball location or version for opensbi doesn't fail. It reverts to
BR2_TARGET_OPENSBI_LATEST_VERSION in that case.

Fixes:
 - http://autobuild.buildroot.org/results/b31882b1a7119b2b590cc6dfed44fe515a63a6b8
 - http://autobuild.buildroot.org/results/61866e8941741c9746dcb2c60b6db3b0e5b9a47b

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-20 11:31:56 +02:00
James Hilliard
69400611b2 utils/scanpypi: restore modules search path in case of error
We extend the modules search path to be able to load the package
metadata. Currently, it is only restored when loading those
succeeded, not when it failed.

Restore it to its previous state also in case of error, to avoid
leaking the path further.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-08-17 21:21:50 +02:00
Fabrice Fontaine
2bf6b06308 utils/genrandconfig: add xenomai custom handling
Add custom cases to make sure that a random configuration with an empty
git, tarball location or version for xenomai doesn't fail. It reverts to
BR2_PACKAGE_XENOMAI_LATEST_VERSION in these cases.

Fixes:
 - http://autobuild.buildroot.org/results/19419759b4add0197b4e629d0b2216c2a07624b2

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-08-15 18:45:00 +02:00
Fabrice Fontaine
3ce97967e0 utils/genrandconfig: fix indentation
Commit 0f3115b2ca broke the indentation
(missing one space)

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-09 13:32:24 +02:00
Fabrice Fontaine
0f3115b2ca utils/genrandconfig: add refpolicy custom git handling
Add a custom case to make sure that a random configuration with an empty
git location for refpolicy doesn't fail. It reverts to
BR2_PACKAGE_REFPOLICY_UPSTREAM_VERSION in that case.

Fixes:
 - http://autobuild.buildroot.org/results/98da07a1f9b88e2e8a028a55c8cf76c6ebc28304

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-08 22:48:29 +02:00
Jesse Taube
b58e2cde5c utils/readme.txt: Fix typo "get-developers"
Fix typo on line 23 `get-developpers` to `get-developers`

Signed-off-by: Jesse Taube <Mr.Bossman075@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-07-23 18:21:21 +02:00
Ricardo Martincoski
7082b0585d utils/get-developers: add -d option for custom DEVELOPERS file
In preparation to the introduction of test cases for the
get-developers script, make it possible to pass a custom DEVELOPERS
file. A normal user of get-developers will most likely never use this,
but the test suite will use it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: extracted from a larger patch from Ricardo, initially posted
at https://patchwork.ozlabs.org/project/buildroot/patch/20220528014832.289907-1-ricardo.martincoski@gmail.com/]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-23 16:32:58 +02:00
Thomas Petazzoni
47f359a615 utils/get-developers: implement a -v action for DEVELOPERS file validation
Commit "45aabcddc5 utils/get-developers: really make it callable from
elsewhere than the toplevel directory" had a unforeseen side-effect:
the fact that check-developers with no arguments used to validate the
syntax of the DEVELOPERS no longer works. Indeed, the
parse_developers() call was moved *after* the check that verifies if
at least one action is passed. Due to this, the check-DEVELOPERS
verification in Gitlab CI, which invokes get-developers without
argument to validate the DEVELOPERS file... no longer does anything
useful.

In order to fix this, we introduce an explicit action in
get-developers to request the validation of the DEVELOPERS file. The
implementation is trivial, as the validation is already done by
calling parse_developers(), but at least now we have an action that
does "nothing", except execute until the validation is done.

This is also much more explicit than "invoke get-developers without
arguments to just do validation".

The initial investigation of the issue was done by Ricardo Martincoski
<ricardo.martincoski@gmail.com> and reported at
https://patchwork.ozlabs.org/project/buildroot/patch/20220528014832.289907-1-ricardo.martincoski@gmail.com/,
but we have chosen to introduce a separate -v option rather than
making the validation part of the existing -c action, which serves a
different purpose.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-23 16:32:58 +02:00
Kory Maincent
d780b828fc utils/genrandconfig: add optee-os custom tarball handling
Add a custom case to make sure that a random configuration with an
empty tarball location for OP-TEE OS doesn't fail. It reverts to
BR2_TARGET_OPTEE_OS_LATEST in that case.

Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-06 23:36:34 +02:00
Thomas Petazzoni
793ee1011e Remove support for the NDS32 architecture
The support for this architecture has been removed from the upstream
Linux kernel, as of commit:

  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aec499c75cf8e0b599be4d559e6922b613085f8f

Which states:

  The nds32 architecture, also known as AndeStar V3, is a custom
  32-bit RISC target designed by Andes Technologies. Support was added
  to the kernel in 2016 as the replacement RISC-V based V5 processors
  were already announced, and maintained by (current or former) Andes
  employees.

  As explained by Alan Kao, new customers are now all using RISC-V,
  and all known nds32 users are already on longterm stable kernels
  provided by Andes, with no development work going into mainline
  support any more.

There has also been little to no maintenance done in Buildroot for
this architecture in recent times, so let's follow the Linux kernel
community decision and drop support for this CPU architecture.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-19 18:30:45 +02:00
James Hilliard
541e794a95 utils/genrandconfig: add ARM trusted firmware fixup
Fixes:
boot/arm-trusted-firmware/arm-trusted-firmware.mk:200: *** No repository specified. Please check BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_REPO_URL.  Stop.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-05-18 21:16:01 +02:00
James Hilliard
fcca757443 utils/genrandconfig: add DTS cuimage fixup
If we don't have a DTS name or path we need to also disable cuimage
to disable DTS support as BR2_LINUX_KERNEL_CUIMAGE requires DTS
support.

Fixes:
linux/linux.mk:591: *** No kernel device tree source specified, check your BR2_LINUX_KERNEL_INTREE_DTS_NAME / BR2_LINUX_KERNEL_CUSTOM_DTS_PATH settings.  Stop.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-05-04 21:38:10 +02:00
James Hilliard
57926b025b utils/genrandconfig: only fixup host uboot tools source when empty
If the kconfig values being replaced are not empty we should not
try to fix them.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-05-03 22:22:04 +02:00
James Hilliard
fe740dcac8 utils/genrandconfig: use high quality random for KCONFIG_SEED
Internally genrandconfig will use gettimeofday when generating a
KCONFIG_SEED, since autobuild-run executes genrandconfig at the same
time for multiple autobuilder runners this could potentially result
in the same KCONFIG_SEED being generated for those test runs started
at the same time.

To ensure this doesn't happen set the KCONFIG_SEED using the urandom
entropy source.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-05-03 22:20:18 +02:00
James Hilliard
71c2a29e5e utils/genrandconfig: fix ext2 config variable
This should have been BR2_TARGET_ROOTFS_EXT2 as the
BR2_TARGET_ROOTFS_EXT2_GEN variable is an integer variable used
to indicate the ext2/ext3/ext4 variant.

Fixes:
 - http://autobuild.buildroot.net/results/5d7/5d7833212bd8a2b575945f848d8c91dabe7d3c30

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-05-02 23:40:23 +02:00
James Hilliard
3c98eb26db utils/genrandconfig: increase default target rootfs size to 5G
Since there isn't a way to reliably compute the required size lets
just set it to what should be a high enough value to not run out
of space.

Both ext2 and fsfs create sparse files, so this is not a waste of disk
space.

Fixes:
 - http://autobuild.buildroot.net/results/a3c68cdc515353c0d2650ee743aa3bd8ab99a418
 - http://autobuild.buildroot.net/results/ad426ff903d78a92de8f0208876e2196136a09a8

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-04-30 20:40:21 +02:00
Thomas Petazzoni
04ea3d38dd arch: drop support for SH2A
Until commit "arch/Config.in.sh: fixup MMU selection" in this series,
SH2A could either be used with BR2_USE_MMU disabled or BR2_USE_MMU
enabled.

The later made absolutely no sense, since SH2A does not have a MMU:
MMU support was introduced starting from SH3 according to
https://en.wikipedia.org/wiki/SuperH#SH-3

Also, since commit 22d5501e03 ("arch:
tidy up binary formats config"), which was merged in Buildroot
2015.05, the architecture tuple used when BR2_sh2a=y and BR2_USE_MMU
disabled is sh2a-buildroot-uclinux-uclibc, and this was already
unsupported back in the days of Buildroot 2015.08 and binutils 2.24,
causing the build to fail with:

*** BFD does not support target sh2a-buildroot-uclinux-uclibc.

just like it fails to build today with recent version of binutils.

So, this has been broken since 2015.08, and nobody complained. SH2A is
seldom used, so it's time to kill it.

It is worth mentioning that there had been an attempt at resurrecting
SH2 support around 2015 (see https://lwn.net/Articles/647636/) as part
of the J2 core. This effort led to the addition of FDPIC support for
SH2A in the musl C library (and therefore proper ELF binaries, with
shared libraries), but that was never supported in Buildroot. Now that
the J2 project is essentially dead, there is no reason to bother with
this.

Fixes:

  http://autobuild.buildroot.net/results/63d01d33ae30f86b63b9f42a9fea116f2f3e9005/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-04-24 10:38:52 +02:00
James Hilliard
aeee90ec10 utils/genrandconfig: add randconfig based --no-toolchains-csv option
Currently we only test a limited set of toolchains that are mostly
prebuilt, add a flag to allow using randconfig for randomizing
additional toolchain settings instead of randpackageconfig.

To avoid invalid configs we need to add additional config validation
filtering and fixups. Although the loop around 'make randconfig'
attempts does make sure we eventually end up with a valid configuration,
there is a bias towards e.g. disabling the kernel. It would be possible
to e.g. always force BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG=y, but
that removes the possibility for a toolchains-csv config to add a valid
custom defconfig. (Note that currently this isn't possible anyway since
'make randpackageconfig' is used if a toolchains-csv is provided, but
eventually we want to use 'make randconfig' also if a toolchains-csv is
provided.)

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-04-09 18:45:03 +02:00
James Hilliard
2b09e05b58 utils/scanpypi: support alternative Homepage format
Some packages only have a home page properly set inside project_urls.

Squelch flake8's E127, because a visual indent here is really nicer.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[yann.morin.1998@free.fr: simplify getting home_page fallbacks]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-03-13 19:24:23 +01:00
Fabrice Fontaine
4dc48e8b9d utils/checkpackagelib/lib_mk.py: check CONF_OPTS
As already done for {FOO}_DEPENDENCIES in commit
4910a175b3, check that {FOO}_CONF_OPTS are
never overridden in a conditional

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-03-07 23:15:46 +01:00
Giulio Benetti
cc9470c2d8 utils/docker-run: hide docker run command
Since this is a helper script there is not much reason to show the
command that's been issued. Furthermore, the incantation has been
slightly extended since the script was introduced.

The only interesting reason to print the command is to know what image
it is being spawned into. However, this is prominently displayed by
docker the first time the script is run, as it can't find the image
locally and has to fetch it first. Afterwards, users can still use
'docker image ls' to see what images they have locally.

So let's remove 'set -x' before running docker.

Signed-off-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
[yann.morin.1998@free.fr: reword and expand commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-02-16 22:55:15 +01:00
Yann E. MORIN
7f2020f904 utils/docker-run: improve user experience
Currently, the docker-run script starts a container based on the
reference image, in which the user has UID:GID set to 1000:1000,
which may or may not be the same as local user uses, which may
prevent the commands in the container from creating files, which
foils the plan of using the container to run a build...

Additionally, the paths in the container differ from the paths
the user expects, as the current Buildroot tree is mounted over
the in-container user home directory. This is a bit confusing...

Finally, the container is left hanging around after the command
finishes, and thus there are as many lingering containers as the
user runs docker-run. This is not very nice...

We fix all of that (yeah, should be different patches, but meh):

  - we use --mount instead of -v, which allows us to bind-mount
    the Buildroot tree at the same place in the container, as
    Docker will create the destination if it does not exist, while
    -v seems to expect it to exist [0].

  - as a consequence, we can also set the working directory as the
    Buildroot top-directory;

  - use --user to force the same UID:GID in the container as the
    local user, so that files created in the container belong to
    the local user, and so that files from the local user are
    accessible from the container;

  - use --rm to remove the container once it terminates; starting
    a new container is very quick anyway, so it is as good as
    re-using a previous container.

[0] the documentation is not clear about that. It clearly states
that the host directory (i.e. the origin, the source) is created
if missing, but it says nothing of the destination:
    https://docs.docker.com/engine/reference/commandline/run/#mount-volume--v---read-only

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Giulio Benetti <giulio.benetti@benettiengineering.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Tested-by: Giulio Benetti <giulio.benetti@benettiengineering.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-16 22:27:24 +01:00
Luca Ceresoli
cef07c9358 utils/docker-run: allow interactive use
Without -i, the container cannot be used interactively. Allow using it as
in:

  $ ./utils/docker-run
  + exec docker run -v /home/ceresoli/devel/buildroot:/home/br-user -it registry.gitlab.com/buildroot.org/buildroot/base:20220206.1756
  br-user@1e9508a4ccbb:~$ ls
  arch  board  boot  CHANGES  Config.in  Config.in.legacy  configs  COPYING  DEVELOPERS  docs  fs  linux	Makefile  Makefile.legacy  output  package  README  support  system  toolchain	utils
  br-user@1e9508a4ccbb:~$

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-02-14 17:56:49 +01:00
Luca Ceresoli
61912cc89a utils/docker-run: use portable bash shebang
/usr/bin/bash does not exist on Ubuntu 2018.04. Use the more portable
alternative "#!/usr/bin/env bash" which we are already using in quite
a few scripts.

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Fixes: 242e9d72e7 ("utils/docker-run: new script")
[yann.morin.1998@free.fr: use "#!/usr/bin/env bash", not "@!/bin/bash"]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-02-14 17:45:16 +01:00
Peter Korsgaard
6c00a319ac utils/genrandconfig: drop BR2_PACKAGE_PYTHON_PY_ONLY logic
The option was removed by commit 306839586b (package/python: drop target
package).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-02-12 21:39:43 +01:00
Arnout Vandecappelle (Essensium/Mind)
b42e87772a utils/checkpackagelib/test_tool.py: fix expectation
While committing the shellcheck feature, it was changed to output the
full shellcheck output even at verbosity level 1. However, the
expectation of the shellcheck test was not updated accordingly.

Do that now, simply merging all the shellcheck output in a single
string.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 19:31:44 +01:00
Ricardo Martincoski
d409fe0856 utils/checkpackagelib/lib_sysv: run shellcheck
For simplicity, when shellcheck returns one or more warnings, count it
as a single check-package warning.
The developer can get the full output either by running shellcheck
directly or by calling check-package with -v.

Examples:

|$ ./utils/docker-run utils/check-package --include Shellcheck package/polkit/S50polkit
|package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings
|51 lines processed
|1 warnings generated

|$ ./utils/docker-run utils/check-package --include Shellcheck -v package/polkit/S50polkit
|package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings
|In package/polkit/S50polkit line 43:
|< tab  >start|stop|restart|reload)
|                           ^----^ SC2221: This pattern always overrides a later one on line 45.
|In package/polkit/S50polkit line 45:
|< tab  >reload)
|        ^----^ SC2222: This pattern never matches because of a previous pattern on line 43.
|For more information:
|  https://www.shellcheck.net/wiki/SC2221 -- This pattern always overrides a l...
|  https://www.shellcheck.net/wiki/SC2222 -- This pattern never matches becaus...
|51 lines processed
|1 warnings generated

NOTICE: shellcheck results depends on the version of the tool.
This is why the examples above run inside the docker image.

Also update .gitlab-ci.yml with the docker image after the change of
the previous commit. We don't actually use shellcheck in CI, but the
image from .gitlab-ci.yml is used by the docker-run script and we use
that to run shellcheck.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: make sure a single -v is enough to get shellcheck output;
 update docker image tag in .gitlab-ci.yml]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 18:27:03 +01:00
Ricardo Martincoski
d020368eea utils/checkpackagelib/lib_sysv: check SysV init scripts
Enable the common checks:
 - consecutive empty lines
 - empty last line
 - missing new line at end of file
 - trailing space
 - warn for executable files, with the hint to instead use
   '$(INSTALL) -D -m 0755' in the .mk file

Check indent with tabs:
 - add a simple check function to warn only when the indent is done
   using spaces or a mix of tabs and spaces. It does not check indenting
   levels, but it already makes the review easier, since it
   diferentiates spaces and tabs.

Check variables:
 - check DAEMON is defined
 - when DAEMON is defined, check the filename is in the form S01daemon
 - when PIDFILE is defined, expect it to be in /var/run and defined
   using $DAEMON.

Also add unit test for this.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: avoid 'del NotExecutable_base' by importing the module instead
 of the class; refer to manual in warnings]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 18:27:00 +01:00
Ricardo Martincoski
7947328de4 utils/checkpackagelib: warn about executable files
Currently there are no .mk, Config.in, .patch or .hash files with
executable permissions in the tree.
But we don't want to have that.

So warn when a file checked by check-package has executable permission.

This check will be reused when testing SysV init scripts in the tree.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: use context manager for temp dir so it gets deleted]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 18:26:20 +01:00
Ricardo Martincoski
1734127e59 utils/check-package: prepare to run external tools
Some file formats have well-established syntax checkers.
One example of this is the tool 'shellcheck' that can analyse shell
scripts for common mistakes.

There is no reason to reimplement such tools in check-package, when we
can just call them.

Add the ability to check-package to call external tools that will run
once for each file to be analysed.
For simplicity, when the tool generated one or more warnings, count it
as a single warning from check-package, that can display something like
this:

|$ ./utils/check-package  package/unscd/S46unscd
|package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings
|25 lines processed
|1 warnings generated

|$ ./utils/check-package -vvvvvvvvvvvvvvvv package/unscd/S46unscd
|package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings
|In package/unscd/S46unscd line 9:
|        printf "Starting ${NAME}: "
|               ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
|In package/unscd/S46unscd line 11:
|        [ $? -eq 0 ] && echo "OK" || echo "FAIL"
|          ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
|In package/unscd/S46unscd line 14:
|        printf "Stopping ${NAME}: "
|               ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
|In package/unscd/S46unscd line 16:
|        [ $? -eq 0 ] && echo "OK" || echo "FAIL"
|          ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
|For more information:
|  https://www.shellcheck.net/wiki/SC2059 -- Don't use variables in the printf...
|  https://www.shellcheck.net/wiki/SC2181 -- Check exit code directly with e.g...
|25 lines processed
|1 warnings generated

In this first commit, add only the ability for check-package to call
external tools and not an example of such tool, as adding each tool to
call may need update to the docker image and can lead to it's own
discussion on how to implement.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 16:38:52 +01:00
Ricardo Martincoski
242e9d72e7 utils/docker-run: new script
Add a small script to run commands in the same docker image used in the
GitLab CI.

For instance, one can run check-package unit tests without installing
pytest directly in the host:
$ ./utils/docker-run python3 -m pytest -v utils/checkpackagelib/

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: fix shellcheck errors; add exec]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 15:35:20 +01:00
Ricardo Martincoski
fc254881e6 utils/checkpackagelib: add unit tests
So anyone willing to contribute to check-package can run all tests in
less than 1 second by using:
$ python3 -m pytest -v utils/checkpackagelib/

Most test cases are in the form:
@pytest.mark.parametrize('testname,filename,string,expected', function)
 - testname: a short description of the scenario tested, added in order
   to improve readability of the log when some tests fail
 - filename: the filename the check-package function being tested thinks
   it is testing
 - string: the content of the file being sent to the function under test
 - expected: all expected warnings that a given function from
   check-package should generate for a given file named filename and
   with string as its content.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 15:35:19 +01:00
Arnout Vandecappelle (Essensium/Mind)
f1768855e7 utils/size-stats-compare: fix flake8 error
utils/size-stats-compare:80:15: E225 missing whitespace around operator

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-01 21:52:35 +01:00
Thomas De Schampheleire
391f70b689 utils/size-stats-compare: add package name in detail output
size-stats-compare gives an overview of the size increase/decrease between
two cases, based on packages-file-list.txt. The 'detail' mode gives info per
file, otherwise per package.

But sometimes, you want the detailed per-file info, but only for a specific
package. Since the detailed output no longer lists the package name, you
cannot simply grep for it. A workaround was to filter the input
packages-file-list.txt's first, and then pass these filtered versions to
size-stats-compare.

Make this easier by adding the package name next to the filename in detailed
output. This allows grep'ing normally.
For example:

  $ utils/size-stats-compare orig new  -t 100 -d | grep ebtables

      -67712 removed ebtables             lib/ebtables/libebtc.so
      -66764 removed ebtables             lib/ebtables/libebt_nat.so
      -66752 removed ebtables             sbin/ebtables
      -66704 removed ebtables             lib/ebtables/libebt_arp.so
      -66700 removed ebtables             lib/ebtables/libebt_stp.so
      -66700 removed ebtables             lib/ebtables/libebt_among.so
      -66684 removed ebtables             lib/ebtables/libebt_ip.so
      -66676 removed ebtables             lib/ebtables/libebt_limit.so
      -66656 removed ebtables             lib/ebtables/libebt_log.so
      -66648 removed ebtables             lib/ebtables/libebt_mark.so
      -66636 removed ebtables             lib/ebtables/libebt_pkttype.so
      -66604 removed ebtables             lib/ebtables/libebt_vlan.so
      -66588 removed ebtables             lib/ebtables/libebt_ulog.so
      -66588 removed ebtables             lib/ebtables/libebt_nflog.so
      -66584 removed ebtables             lib/ebtables/libebt_arpreply.so
      -66544 removed ebtables             lib/ebtables/libebt_ip6.so
      -66540 removed ebtables             lib/ebtables/libebt_802_3.so
      -66536 removed ebtables             lib/ebtables/libebt_standard.so
      -66524 removed ebtables             lib/ebtables/libebt_mark_m.so
      -66524 removed ebtables             lib/ebtables/libebt_redirect.so
      -66452 removed ebtables             lib/ebtables/libebtable_filter.so
      -66452 removed ebtables             lib/ebtables/libebtable_broute.so
      -66452 removed ebtables             lib/ebtables/libebtable_nat.so
          45         ebtables             etc/ethertypes
       66752 added   ebtables             usr/sbin/ebtablesd
       66752 added   ebtables             usr/sbin/ebtables-legacy
       66752 added   ebtables             usr/sbin/ebtablesu
      200840 added   ebtables             usr/lib/libebtc.so.0.0.0

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
2022-01-31 21:32:22 +01:00
Thomas De Schampheleire
e9b1dffa5d utils/size-stats-compare: clarify meaning of variables in print_result
print_result is juggling with entry[x][y] which is not very readable.
While a better solution would be to use a class and reference named
attributes, that would require some bigger changes in the script.

Instead, make a minimal improvement by assigning the entry[x][y] values to
intermediate variables. Store them in a dict for easy usage from a format
string.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-31 21:24:40 +01:00
Yann E. MORIN
5da8218184 support/download: drop support for the 'none' hash
Commit 1ba85b7f87 (support/download: add explicit no-hash support)
introduced the 'none' hash type, in an attempt to make hash files
mandatory, but not failing on archives localy generated, like those
for git or svn repositories, especially for those packages where a
version choice was present, which would allow for either remote
archives for which we'd have a hash or VCS trees for which we could
not have a hash for the localy generated archive.

Indeed, back in the time, we did not have a mean to generate
reproducible archives, so having a hash file without a hash for
thosel ocally generated archives would trigger an error in the
hash-checking machinery.

But now, low-and-behold, we do know how to generate those archives,
and we have a mechanism to explicitly exclude some archives from being
hash-checked (e.g. when the version string itself can be user-provided).

As such, the 'none' hash type no longer has any raison d'être, we do not
use it in-tree, and its use in a br2-external tree is most probably
inexistent (as is the use of hash files alotgether most probably).

So we simply drop the support for that.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Thomas: drop support in checkpackagelib, as reported by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-01-11 21:58:01 +01:00
Yann E. MORIN
f29239cf9b utils/test-pkg: add check that show-info is proper json
JSON parser are usually rather strict when parsing their input. Both
jq and the python json module choke on non-conformant inputs.

Commit e4c284e6b9 (package/pkg-utils: escape \ in generated legal-info)
fixed the \-escaping case, but we want to ensure that we do not have
other data that would be improperly encoded.

To more easily catch issues, also run show-info and send its output
through jq, to try and validate the output.

Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-10 08:52:58 +01:00
Heiko Thiery
f35a4b4ae2 utils/check-package: add a check for the new spacing convention
The seperation of the fields in the hash file should be 2 spaces for
consistency.

Since a large number of hash files still violate this rule, exclude it
from "make check-package" (and thus from CI).

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout:
 - Move it to a separate class, so it can be excluded.
 - Exclude it from "make check-package"
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-09 23:01:53 +01:00
Arnout Vandecappelle (Essensium/Mind)
50b8d6307f utils/diffconfig: use python3 explicitly
Python 2 is EOL sice 2020 [1], it's still available on distros, but may
not be installed by default (as being replaced by python3). A
compatibility symlink python -> python3 may not be installed either.

Convert the shebang line to the usual /usr/bin/env based line.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-12-29 10:07:59 +01:00
Ricardo Martincoski
29bb478a49 utils/getdeveloperlib.py: fix developer being reported for unrelated path
Currently, by following the instructions in the manual and querying for
developers for a patch that changes path
package/foobar
the script reports both developers that have these entries in the
DEVELOPERS file:
F:	package/foo/
F:	package/foobar/

Starting from commit "afc112b0e4 utils/getdeveloperlib.py: fix issue
with hasfile()" get-developers script uses os.path.abspath() and
os.path.relpath().
The catch is that those functions return the absolute path and the
relative path without the trailing slash.

When the paths associated to a developer are then compared to the paths
a patch touches, using the string.startswith(), any substring returns
True, leading to developers for package/foo/ being wrongly reported
for package/foobar/ .

Fix this by re-adding the trailing slash after using relpath().

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: James Knight <james.d.knight@live.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-11 21:00:29 +01:00
Ricardo Martincoski
5bbedea9c2 utils/checkpackagelib/lib_mk.py: fix check for overridden variable
Currently this .mk snippet results in unexpected behavior from
check-package:
|VAR_1 = VALUE1
|ifeq (condition)
|VAR_1 := $(VAR_1), VALUE2
|endif

Fix commit "163f160a8e utils/{check-package, checkpackagelib}:
consistently use raw strings for re.compile" that ended up doing this:
-    CONCATENATING = re.compile("^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")
+    CONCATENATING = re.compile(r"^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")

But raw strings do not expect escaping when referencing \1 and the
pattern ends up searching for a raw '\\1' instead of an occurrence of
the first pattern inside parenthesis.

|$ python3
|Python 3.8.10 (default, Sep 28 2021, 16:10:42)
|[GCC 9.3.0] on linux
|Type "help", "copyright", "credits" or "license" for more information.
|>>> import re
|>>> p1 = re.compile('(foo)bar\\1')
|>>> p2 = re.compile(r'(foo)bar\\1')
|>>> p3 = re.compile(r'(foo)bar\1')
|>>> s1 = 'foobarfoo'
|>>> s2 = 'foobar\\1'
|>>> print(p1.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p2.search(s1))
|None
|>>> print(p3.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p1.search(s2))
|None
|>>> print(p2.search(s2))
|<re.Match object; span=(0, 8), match='foobar\\1'>
|>>> print(p3.search(s2))
|None
|>>>

So use '\1' instead of '\\1' in the raw string.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-10 20:01:40 +01:00
Fabrice Fontaine
4910a175b3 utils/checkpackagelib/lib_mk.py: check DEPENDENCIES
Check that {FOO}_DEPENDENCIES are never overriden in a conditional

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-10 19:26:30 +01:00
Fabrice Fontaine
fba568a16e utils/checkpackagelib/lib_mk.py: check typo in define
Check typo in define to detect SMAKE_LINUX_CONFIG_FIXUPS in smack
(fixed by 41e2132fbe)

The new expression will catch "SMAKE_CONF_OPTS" as well as
"define SMAKE_LINUX_CONFIG_FIXUPS"

Two modifications were made:
 - add (define\s+)? which will match "define " but also an empty value.
   Thanks to this, the second group will always contain the variable
   name.
 - remove \s*(\+|)= which seems superfluous

Also, add GCC_TARGET in ALLOWED variable to avoid the following
warnings:

arch/arch.mk:12: possible typo: GCC_TARGET_ARCH -> *ARCH*
arch/arch.mk:13: possible typo: GCC_TARGET_ABI -> *ARCH*
arch/arch.mk:14: possible typo: GCC_TARGET_NAN -> *ARCH*
arch/arch.mk:15: possible typo: GCC_TARGET_FP32_MODE -> *ARCH*
arch/arch.mk:16: possible typo: GCC_TARGET_CPU -> *ARCH*
arch/arch.mk:17: possible typo: GCC_TARGET_FPU -> *ARCH*
arch/arch.mk:18: possible typo: GCC_TARGET_FLOAT_ABI -> *ARCH*
arch/arch.mk:19: possible typo: GCC_TARGET_MODE -> *ARCH*

Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-06 22:12:34 +01:00
James Hilliard
c3029878c5 utils/scanpypi: fix buildutils check in python3
The message attribute does not exist in python3, see PEP-0352:
https://www.python.org/dev/peps/pep-0352/

Fixes:
Traceback (most recent call last):
  File "utils/scanpypi", line 743, in <module>
    main()
  File "utils/scanpypi", line 693, in main
    if 'buildutils' in err.message:
AttributeError: 'ImportError' object has no attribute 'message'

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-06 22:12:11 +01:00