Commit Graph

307 Commits

Author SHA1 Message Date
Bernd Kuhls
9b31b64382 utils/genrandconfig: increase default target ubifs image size
The new value is calculated based on:
ROOTFS_SIZE = 5G (see commit 3c98eb26db)
BR2_TARGET_ROOTFS_UBIFS_LEBSIZE = 0x1f800

5G / 0x1f800 = 41610

Fixes:
http://autobuild.buildroot.net/results/f72/f72918d63510b170e5da01bfa9c247cf9dcf507f/

Signed-off-by: Bernd Kuhls <bernd@kuhls.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-07-14 23:26:09 +02:00
Bernd Kuhls
af5ccec266 utils/genrandconfig: disable cramfs images
Fixes build error:

  warning: estimate of required size (upper bound) is 1374MB, but
  maximum image size is 272MB, we might die prematurely
  mkcramfs: filesystem too big

Fixes:
http://autobuild.buildroot.net/results/d47/d47f9b462707dffe1b6665f143701303b04e2adc/

Signed-off-by: Bernd Kuhls <bernd@kuhls.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-07-09 18:30:10 +02:00
Yann E. MORIN
791c163b2f utils/docker-run: make it work in workdirs/woktrees
It is quite customary to use a single repository with multiple workdirs,
one for each active branch, with either the aging 'git new-workdir' or
the more recent 'git worktree'.

However, in a workdir/worktree, most entries in .git/ are only symlinks
to the actual files in the main repository.

Currently, utils/docker-run only bind-mounts the current working copy.
If that is a workdir/worktree, then it is going to be missing the actual
git data, resulting in errors like:

    $ ./utils/docker-run make check-package
    fatal: not a git repository (or any parent up to mount point [....]/buildroot)
    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
    No files to check style
    make: *** [Makefile:1257: check-package] Error 1

So, we also bind-mount the actual git directory. If that is a subdir
of the current working copy, then it is already mounted and thus the
bind-mount is superfluous but harmless; for simplicity, we mount it
unconditionally.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
2023-05-13 12:09:32 +02:00
Yann E. MORIN
5eac4f81eb utils/check-package: check ignored files exist
When an ignored file is removed (e.g. a package patch is no longer
needed after a version bump), the corresponding entry in the ignore list
is no longer needed.

However, we currently only validate that an ignored *test* still fails,
not that a ignore files is now missing.

Add a new test to check-package that does that check, and add a
test-case for that check.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
2023-05-13 12:08:25 +02:00
Yann E. MORIN
a3e796b74e utils/check-package: fix grammar
The past participle for "to fix" is "fix". The "did you forget" got
eluded into "forget", so again a past participle.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-04-23 21:49:46 +02:00
Yann E. MORIN
3d8212c4b2 utils/docker-run: allow running without a tty
Currently, utils/docker-run spawns a container with a tty, so that he
user can interact properly in the container.

However, that requires a tty when calling docker-run, which is not
always guaranteed, e.g. if called from a git hook.

Since the script is a bash script already, we can use an array to store
options passed to docker, and only add the -t option when there is
actually a tty available.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
2023-04-23 19:34:00 +02:00
Vincent Fazio
32934b526b utils/checkpackagelib: check for Upstream trailers
Implement a check-package check for an Upstream: trailer in patches
being applied to packages per a mailing list discussion [0].

No strict formatting checks are implemented for the contents within the
trailer as the needed level of detail will vary patch-to-patch.

Tested with: `./utils/docker-run python3 -m pytest utils/checkpackagelib`

[0] https://lists.buildroot.org/pipermail/buildroot/2023-March/666016.html

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-04-15 19:36:50 +02:00
Vincent Fazio
b7d4ae98fa utils/check-package: drop six usage
The shebang in check-package now defines python3. There is no longer a
need to maintain support with python 2.x.

See-also: 02b165dc71 (check-package: fix Python3 support)

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
Tested-by: James Knight <james.d.knight@live.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-04-10 21:02:35 +02:00
Ricardo Martincoski
29a0dd4a30 utils/checkpackagelib: warn about $(HOST_DIR)/usr
It's been ages (5 years at the next release) that we've not installed
host packages in $(HOST_DIR)/usr, but we still have a few packages that
reference it or install things in there. See [1]

Add a new check_function that warns when a file is added installing to
or referencing $(HOST_DIR)/usr .

[1] "d9ff62c4cd pacakge: drop remnants of $(HOST_DIR)/usr"

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: exclude skeleton.mk with disable comment instead of explicit
         code]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-04-09 22:56:16 +02:00
Ricardo Martincoski
9df6503ed0 Makefile: merge check-flake8 into check-package
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)

Expand the runtime test and the unit tests for check-package.

Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.

Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-04-09 22:33:24 +02:00
Ricardo Martincoski
60fdaf56fe utils/check-package: check files in support/
The .mk files inside both support/dependencies and support/misc are not
package recipes, similar to package/pkg-*.mk. The check-package don't
apply to them. Therefore ignore such files.

In the test infra, some br2-externals are used as fixtures to provide
(sometimes) failure cases, so ignore files in these directories.

Files inside support/kconfig are files copied from linux upstream, so do
not generate warnings for them.

support/gnuconfig contains auto-generated config.{guess,sub} files,
so do not generate shellcheck warnings for them.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-04-09 21:59:04 +02:00
Ricardo Martincoski
29d6f319a0 utils/check-package: check files in board/
When a SysV init script is inside package/ it doesn't need to be
executable. However, when an init script is inside a fs_overlay, it
*does* need to be executable. Therefore, skip the NotExecutable test for
init scripts. We detect them based on the directory /etc/init.d

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: update .checkpackageignore]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-04-09 21:51:11 +02:00
Fabrice Fontaine
ad4982d18e utils/genrandconfig: add boot-wrapper-aarch64 handling
Fix the following build failure raised since the addition of the package
in commit 7689b72e00:

configure: error: Could not find DTB file: /home/autobuild/autobuild/instance-9/output-1/build/linux-6.1.9/arch/arm64/boot/dts/.dtb

Fixes:
 - http://autobuild.buildroot.org/results/44287ccc8cc9767704642919e6d928d1f57b436d

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-27 16:22:54 +01:00
Fabrice Fontaine
5e1a50e4e6 utils/genrandconfig: add lpc32xxcdl handling
Fix the following build failure raised since the addition of the package
in commit 20695936ad:

make[2]: *** /tmp/instance-0/output-1/build/lpc32xxcdl-2.11/csps/lpc32xx/bsps//source: No such file or directory.  Stop.

Fixes:
 - http://autobuild.buildroot.org/results/72c9a4080318b1b247ca3517c95c689dff1068d2

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-25 21:53:48 +01:00
Fabrice Fontaine
02b6af2663 utils/genrandconfig: add ti-k3-r5-loader handling
Fix the following build failure raised since the addition of the package
in commit 0189bcb47c:

make[2]: *** No rule to make target '_defconfig'.  Stop.

Fixes:
 - http://autobuild.buildroot.org/results/d7449b2b2f2349af672bfeee832b89a223a7d9cc

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-25 10:39:05 +01:00
Ricardo Martincoski
2b021bc1d7 utils/check-package: check files in utils/
All the errors in existing scripts in utils/ have been fixed, so nothing
needs to be added to .checkpackageignore.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 15:28:33 +01:00
Arnout Vandecappelle
dab7a87714 utils/docker-run: fix shellcheck errors
In utils/test-pkg line 8:
    if [ ! -z "${TEMP_CONF}" ]; then
         ^-- SC2236: Use -n instead of ! -z.

In utils/test-pkg line 75:
        TEMP_CONF=$(mktemp /tmp/test-${pkg}-config.XXXXXX)
                                     ^----^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        TEMP_CONF=$(mktemp /tmp/test-"${pkg}"-config.XXXXXX)

In utils/test-pkg line 76:
        echo "${pkg_br_name}=y" > ${TEMP_CONF}
                                  ^----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "${pkg_br_name}=y" > "${TEMP_CONF}"

In utils/test-pkg line 86:
    if [ ${random} -gt 0 ]; then
         ^-------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    if [ "${random}" -gt 0 ]; then

In utils/test-pkg line 90:
    if [ ${number} -gt 0 ]; then
         ^-------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    if [ "${number}" -gt 0 ]; then

In utils/test-pkg line 109:
    toolchains=($(sed -r -e 's/,.*//; /internal/d; /^#/d; /^$/d;' "${toolchains_csv}" \
                ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).

In utils/test-pkg line 110:
                  |if [ ${random} -gt 0 ]; then \
                        ^-------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
                  |if [ "${random}" -gt 0 ]; then \

In utils/test-pkg line 111:
                      sort -R |head -n ${random}
                                       ^-------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
                      sort -R |head -n "${random}"

In utils/test-pkg line 121:
    if [ ${nb_tc} -eq 0 ]; then
         ^------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    if [ "${nb_tc}" -eq 0 ]; then

In utils/test-pkg line 134:
        printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
                                                          ^---^ SC2086: Double quote to prevent globbing and word splitting.
                                                                ^------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} "${nb}" "${nb_tc}"

In utils/test-pkg line 146:
        ${nb} ${nb_skip} ${nb_fail} ${nb_legal} ${nb_show}
        ^---^ SC2086: Double quote to prevent globbing and word splitting.
              ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                         ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                                    ^---------^ SC2086: Double quote to prevent globbing and word splitting.
                                                ^--------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        "${nb}" "${nb_skip}" "${nb_fail}" "${nb_legal}" "${nb_show}"

In utils/test-pkg line 160:
    CONFIG_= support/kconfig/merge_config.sh -O "${dir}" \
            ^-- SC1007: Remove space after = if trying to assign a value (for empty string, use var='' ... ).

In utils/test-pkg line 181:
    if [ ${prepare_only} -eq 1 ]; then
         ^-------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    if [ "${prepare_only}" -eq 1 ]; then

For more information:
  https://www.shellcheck.net/wiki/SC1007 -- Remove space after = if trying to...
  https://www.shellcheck.net/wiki/SC2207 -- Prefer mapfile or read -a to spli...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

The suggestions from shellcheck can be applied.

This script already uses bash so we can rely on mapfile.

The warning about CONFIG_= assignment misinterpreted the intention: we
don't want to assign to CONFIG_, we want to clear it from the
environment. Spell this as CONFIG_="".

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 15:28:32 +01:00
Arnout Vandecappelle
2c3466bbee utils/docker-run: fix shellcheck errors
In utils/docker-run line 10:
    --user $(id -u):$(id -g) \
           ^------^ SC2046: Quote this to prevent word splitting.
                    ^------^ SC2046: Quote this to prevent word splitting.

The suggestions from shellcheck can be applied.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 15:28:31 +01:00
Arnout Vandecappelle
7150edc790 utils/config: fix shellcheck errors
In utils/config line 54:
                ARG="`echo $ARG | tr a-z- A-Z_`"
                     ^------------------------^ SC2006: Use $(...) notation instead of legacy backticked `...`.
                           ^--^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
                ARG="$(echo "$ARG" | tr a-z- A-Z_)"

In utils/config line 87:
        local tmpfile="$infile.swp"
              ^-----^ SC2034: tmpfile appears unused. Verify use (or export if used externally).

In utils/config line 182:
                        if [ $? != 0 ] ; then
                             ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- tmpfile appears unused. Verify us...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
  https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...

The suggestions from shellcheck can be applied.

The unused variable tmpfile in fact occurs in several functions, all of
them can be removed.

For the check exit code, the condition is swapped to avoid negative
logic.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 15:28:30 +01:00
Arnout Vandecappelle
dc364c6ae6 utils/brmake: fix shellcheck errors
In utils/brmake line 6:
    local found ret start d h m mf
          ^---^ SC2034: found appears unused. Verify use (or export if used externally).

In utils/brmake line 16:
    > >( while read line; do
               ^--^ SC2162: read without -r will mangle backslashes.

For both, the suggestions from shellcheck can be applied.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 15:28:28 +01:00
Ricardo Martincoski
c5a3dfb9a4 utils/check-package: check all shell scripts
Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).

Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.

In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.

Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.

Apply these checks for shell scripts:
 - shellcheck;
 - trailing space;
 - consecutive empty lines;
 - empty last line on file;
 - newline at end of file.

Update the list of ignored warnings.

Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-08 13:31:06 +01:00
Sébastien Szymanski
09ac5a9e18 utils/test-pkg: proper output on host without jq installed
On host without jq installed, test-pkg's output is as following:

$ ./utils/test-pkg -p mmc-utils
                    bootlin-armv5-uclibc [1/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
JQ IS
OK
                     bootlin-armv7-glibc [2/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
JQ IS
OK
                   bootlin-armv7m-uclibc [3/6]: which: no jq in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/lib64/opencascade/bin)
...

Since test-pkg handles this case we can hide this error message and have
proper output:

$ ./utils/test-pkg -p mmc-utils
                    bootlin-armv5-uclibc [1/6]: OK
                     bootlin-armv7-glibc [2/6]: OK
                   bootlin-armv7m-uclibc [3/6]: OK

Signed-off-by: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-02-07 17:48:42 +01:00
Ricardo Martincoski
de03312db0 utils/get-developers: print error for correct line
Start counting the line numbers in 1 instead of 0, in case an error
must be printed.

Both the error about a developer entry with no file entry and the error
about a file entry with no developer entry actually belong to the
non-empty line previous the one being analysed, so in these cases print
the line number from the line before.

Also count empty and comment lines, so a developer fixing the file can
jump to the correct line (or the nearest one).

At same time standardize the messages, printing the line number
also in the case of a warning for a file that is not in the tree
anymore.

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:24:16 +01:00
Ricardo Martincoski
5ee1dd85a3 utils/get-developers: bail out on parsing errors
Currently 4 types of parsing errors/warnings can be found:
- entry for a file that is not in the tree anymore (warning)
- developer entry with no file entry (error)
- file entry with no developer (error)
- entry that is not a developer, a file or a comment (hard error)

Currently only the last one ends the script with -v with error code.

Make all 3 error types into hard errors and bail out at the first error
found, because the rest of the state machine is not designed to handle
malformed input.
Suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 09:24:16 +01:00
Ricardo Martincoski
dd28acc1e8 Makefile: add target to update .checkpackageignore
When a developer fixes an ignored warning from check-package, he/she
needs to update .checkpackageignore
By running './utils/docker-run make check-package' the developer
receives a warning about this.
Make that change easier to make, by adding a helper target on Makefile.

Add an option --failed-only to check-package that generates output in
the format:
<filename> <check_function> [<check_function> ...]
This is the very same format used by check-package ignore file.

Add the phony target .checkpackageignore

So one can update the ignore file using:
$ ./utils/docker-run make .checkpackageignore

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 18:54:18 +01:00
Ricardo Martincoski
ddf4586b00 utils/check-package: decouple adding rules from fixing all intree files
When a new check_function is added to check-package, often there are
files in the tree that would generate warnings.

An example is the Sob check_function for patch files:
| $ ./utils/check-package --i Sob $(git ls-files) >/dev/null
| 369301 lines processed
| 46 warnings generated
Currently these warnings are listed when calling check-package directly,
and also at the output of pkg-stats, but the check_function does not run
on 'make check-package' (that is used to catch regressions on GitLab CI
'check-package' job) until all warnings in the tree are fixed.
This (theoretically) allows new .patch files be added without SoB,
without the GitLab CI catching it.

So add a way to check-package itself ignore current warnings, while
still catching new files that do not follow that new check_function.

Add a file named .checkpackageignore to the buildroot topdir.
It contains the list of check_functions that are expected to fail for
each given intree file tested by check-package.
Each entries is in the format:
<filename> <check_function> [<check_function> ...]

These are 2 examples of possible entries:
package/initscripts/init.d/rcK ConsecutiveEmptyLines EmptyLastLine Shellcheck
utils/test-pkg Shellcheck

Keeping such a list allows us to have fine-grained control over which
warning to ignore.

In order to avoid this list to grow indefinitely, containing entries for
files that are already fixed, make each entry an 'expected to fail'
instead of just an 'ignore', and generate a warning if a check_function
that was expect to fail for a given files does not generate that
warning.
Unfortunately one case that do not generate warning is an entry for a
file that is deleted in a later commit.

By default, all checks are applied. The --ignore-list option allows to
specify a file that contains the list of warnings that should be
ignored.

The paths in the ignore file must be relative to the location of the
ignore file itself, which means:

 - in the main Buildroot tree, the paths in the ignore file are
   relative to the root of the main Buildroot tree

 - in a BR2_EXTERNAL tree, if the ignore file is at the root of the
   BR2_EXTERNAL, the paths it contains must be relative to that root
   of the BR2_EXTERNAL

This is one more step towards standardizing the use of just 'make
check-package' before submitting patches to the list.

Cc: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-06 18:54:18 +01:00
Ricardo Martincoski
0c5472ace2 utils/check-symbols: new script
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .

The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).

Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.

At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
  (for legacy handling) but is referenced in the package without a
  comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
  but it is not actually referenced.

There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.

Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'

At same time the script is created:
- add unit tests for it, they can be run using:
  utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
  using the script) and check-check-symbols (to check the script against
  its unit tests)

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
	change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 16:30:20 +01:00
James Hilliard
579b65b552 utils/genrandconfig: migrate to asyncio subprocess calls
Since genrandconfig no longer appears to support python2 we can
migrate the subprocess calls to use asyncio variants.

This has the advantage of allowing for runners like autobuild-run to
integrate directly into genrandconfig by calling the asyncio
gen_config using importlib instead of having to run genrandconfig as
a subprocess.

Using asyncio is advantageous here as it eliminates the requirement
for the runner to deal with blocking subprocess calls(by having to
use threading for example).

Also cleanup some unused functions/python2 compatibility shims.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2023-02-06 15:54:38 +01:00
Ricardo Martincoski
29c9b44355 utils/checkpackagelib: warn about ifdef on .mk
There are two legitimate cases to prefer ifdef over ifeq in package
recipes: command-line overrides are allowed for busybox and uclibc
configs.

Except for that, all package in tree already use ifeq, so warn the
developer adding/changing a package to use ifeq instead of ifdef, in
order to keep consistence across packages.
file.mk:2: use ifeq ($(SYMBOL),y) instead of ifdef SYMBOL
file.mk:5: use ifneq ($(SYMBOL),y) instead of ifndef SYMBOL

The difference between ifeq and ifdef is that ifdef doesn't expand
recursively.

Add comments to busybox and uclibc packages to avoid a warning in such
special cases.

Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 14:22:38 +01:00
Ricardo Martincoski
4bbda72743 utils/checkpackagelib: warn about redefined config
Warn the developer in the case the same config is declared more than
once in the same Config.in file.
But take into account the conditional code that lets the config be
visible and warn only when it is declared more than once in the same
conditions.
For instance, do not warn for:
 if BR2_PACKAGE_BUSYBOX
 config BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
 endif
 if !BR2_PACKAGE_BUSYBOX # kconfig doesn't support else
 config BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
 endif

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-06 11:26:41 +01:00
Francois Perrad
9103c8f2e9 package/perl: bump to version 5.36.0
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-01-12 20:33:22 +01:00
Yann E. MORIN
4cbef389f3 utils/scancpan: bump required perl version
As the comment in package/perl/perl.mk instructs, bumping the perl
version must be propagated to utils/scancpan as well.

However, commit 7c1ef8129f (package/perl: bump to version 5.34.0)
forgot to do so.

Fixes: 7c1ef8129f

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-12-20 17:22:27 +01:00
Peter Seiderer
072df09e2f utils/scancpan: update cpan.org and spdx.org URLs from http to https
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-12-20 17:12:19 +01:00
Fabrice Fontaine
5c1f67428a utils/genrandconfig: add mxs-bootlets board handling
Add a custom case to make sure that a random configuration with an empty
board for mxs-bootlets doesn't fail. It reverts to
BR2_TARGET_MXS_BOOTLETS_STMP37xx in that case.

>>> mxs-bootlets 10.12.01 Building
BOARD= CROSS_COMPILE="/home/thomas/autobuild/instance-1/output-1/per-package/mxs-bootlets/host/bin/arm-buildroot-linux-uclibcgnueabi-" /usr/bin/make -j1 -C /home/thomas/autobuild/instance-1/output-1/build/mxs-bootlets-10.12.01 power_prep
/home/thomas/autobuild/instance-1/output-1/per-package/xinetd/host/bin/arm-buildroot-linux-uclibcgnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0  -static -DNO_RPC  -I../../include   -c -o inet_aton.o inet_aton.c
make[1]: Entering directory '/home/thomas/autobuild/instance-1/buildroot'
make[1]: warning: -j1 forced in submake: resetting jobserver mode.
build power_prep
/usr/bin/make -C power_prep ARCH= BOARD=
make[2]: Entering directory '/home/thomas/autobuild/instance-1/output-1/build/mxs-bootlets-10.12.01/power_prep'
/usr/bin/make -C ./../mach-/hw
make[3]: Entering directory '/home/thomas/autobuild/instance-1/output-1/build/mxs-bootlets-10.12.01/power_prep'
make[3]: *** ../mach-/hw: No such file or directory.  Stop.

Fixes:
 - http://autobuild.buildroot.org/results/44a2efc64b9b8ff4541430d6b649e7a11a4e4873

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2022-12-05 08:14:00 +01:00
Yegor Yefremov
ecc33ec02a utils/scanpypi: add LICENCE.TXT to the list of the license files
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-23 23:20:20 +01:00
Thomas Petazzoni
75cb8a4902 utils/genrandconfig: don't build igh-ethercat drivers
igh-ethercat comes with a small number of patched Linux kernel network
drivers, which aim at replacing the ones available in upstream Linux
kernel. All those drivers are provided only for specific kernel
releases. For example:

r8169-2.6.24-ethercat.c
r8169-2.6.24-orig.c
r8169-2.6.27-ethercat.c
r8169-2.6.27-orig.c
r8169-2.6.28-ethercat.c
r8169-2.6.28-orig.c
r8169-2.6.29-ethercat.c
r8169-2.6.29-orig.c
r8169-2.6.31-ethercat.c
r8169-2.6.31-orig.c
r8169-2.6.32-ethercat.c
r8169-2.6.32-orig.c
r8169-2.6.33-ethercat.c
r8169-2.6.33-orig.c
r8169-2.6.35-ethercat.c
r8169-2.6.35-orig.c
r8169-2.6.36-ethercat.c
r8169-2.6.36-orig.c
r8169-2.6.37-ethercat.c
r8169-2.6.37-orig.c
r8169-3.10-ethercat.c
r8169-3.10-orig.c
r8169-3.12-ethercat.c
r8169-3.12-orig.c
r8169-3.14-ethercat.c
r8169-3.14-orig.c
r8169-3.16-ethercat.c
r8169-3.16-orig.c
r8169-3.2-ethercat.c
r8169-3.2-orig.c
r8169-3.4-ethercat.c
r8169-3.4-orig.c
r8169-3.6-ethercat.c
r8169-3.6-orig.c
r8169-3.8-ethercat.c
r8169-3.8-orig.c
r8169-4.4-ethercat.c
r8169-4.4-orig.c

Obviously, this doesn't play well with the random configuration
testing done by utils/genrandconfig. This commit avoids this issue by
making sure we never build any of those drivers as part of the
genrandconfig generated configurations.

Fixes:

  http://autobuild.buildroot.net/results/07b7475d780c067d99ee5618a5fd2bb024a5b4e7/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-11-21 22:20:43 +01:00
Thomas Petazzoni
c81da5e41c utils/genrandconfig: disallow configs with BR2_XTENSA_CUSTOM=y
When BR2_XTENSA_CUSTOM=y is used with the internal toolchain, an
overlay file is mandatory, which genrandconfig can't provide. So we
simply disallow such configurations.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-11-13 22:37:27 +01:00
Fabrice Fontaine
bada3ae53f utils/genrandconfig: handle a10disp
a10disp will raise the following build failure with a mainline kernel:

cp: cannot stat '/home/autobuild/autobuild/instance-0/output-1/build/linux-5.17.15/include/video/sunxi_disp_ioctl.h': No such file or directory

Fixes:
 - http://autobuild.buildroot.org/results/1f2607d6adece4d5dfe17fbdb032a2d228fc030f

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-10-31 13:20:03 +01:00
Thomas Petazzoni
e1550ef755 toolchain/toolchain-buildroot: introduce BR2_TOOLCHAIN_BUILDROOT_NONE
In the internal toolchain backend, we have a choice..endchoice block
to allow the user to select the C library, between glibc, uClibc and
musl.

However, there are situations were no C library at all is
supported. In this case, the choice does not appear, and does not
allow to see the Config.in comments that are within the
choice..endchoice block and that may explain why no C library is
available.

For example, on RISC-V 32-bit, the only C library supported is glibc,
and the minimum kernel header version required by glibc on this
architecture is 5.4.0. In a future commit, we are going to add this
dependency on glibc (to fix build issues on configurations that have
headers < 5.4.0). But since glibc is the only supported C library on
RISC-V 32-bit, it means that the choice..endchoice for the C library
contains no entry, preventing from seeing the Config.in comment.

To address this issue, this commit adds a "dummy"
BR2_TOOLCHAIN_BUILDROOT_NONE option that shows up in the
choice..endchoice only when no C library is available. Thanks to this,
the choice..endchoice is never empty, and the Config.in comments can
be seen.

If the user keeps BR2_TOOLCHAIN_BUILDROOT_NONE selected, then the
build will anyway abort early because package/Makefile.in has a check
to verify that a C library is selected, and aborts the build if not.

Some could say that the problem should be resolved by instead
preventing the selection of headers < 5.4.0 on RISC-V 32-bit, but that
is difficult to do as the user can choose a custom header version, or
simply specific that (s)he wants to use the headers of the kernel
being built. In those situations, it's difficult to prevent selecting
headers < 5.4.0.

Prevent random configurations from triggering a build failure in our
autobuilders, by excluding that symbol from accepted configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: update genrandconfig]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-10-30 12:42:41 +01:00
Yann E. MORIN
fa538315dc utils/genrandconfig: fix flake8
Commit af494d92d3 (utils/genrandconfig: disable libopenssl without
atomics) intorduced a code-style issue that flake8 does not like:

    $ make check-flake8
    utils/genrandconfig:253:8: E713 test for membership should be 'not in'
    1     E713 test for membership should be 'not in'

Fixes: af494d92d3
    https://gitlab.com/buildroot.org/buildroot/-/jobs/3045260108

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-09-18 17:09:30 +02:00
Fabrice Fontaine
af494d92d3 utils/genrandconfig: disable libopenssl without atomics
libopenssl needs atomic or the build will fail (e.g. on sparcv8 without
libatomic):

${LDCMD:-/nvmedata/autobuild/instance-7/output-1/host/bin/sparc-buildroot-linux-uclibc-gcc}  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O0 -g2  -g2  -L.   \
	-o apps/openssl apps/asn1pars.o apps/ca.o apps/ciphers.o apps/cms.o apps/crl.o apps/crl2p7.o apps/dgst.o apps/dhparam.o apps/dsa.o apps/dsaparam.o apps/ec.o apps/ecparam.o apps/enc.o apps/engine.o apps/errstr.o apps/gendsa.o apps/genpkey.o apps/genrsa.o apps/nseq.o apps/ocsp.o apps/openssl.o apps/passwd.o apps/pkcs12.o apps/pkcs7.o apps/pkcs8.o apps/pkey.o apps/pkeyparam.o apps/pkeyutl.o apps/prime.o apps/rand.o apps/rehash.o apps/req.o apps/rsa.o apps/rsautl.o apps/s_client.o apps/s_server.o apps/s_time.o apps/sess_id.o apps/smime.o apps/speed.o apps/spkac.o apps/srp.o apps/storeutl.o apps/ts.o apps/verify.o apps/version.o apps/x509.o \
	 apps/libapps.a -lssl -lcrypto -ldl
/nvmedata/autobuild/instance-7/output-1/host/lib/gcc/sparc-buildroot-linux-uclibc/10.3.0/../../../../sparc-buildroot-linux-uclibc/bin/ld: ./libssl.so: undefined reference to `__atomic_fetch_sub_4'

It should be noted that openssl3 has added OPENSSL_DEV_NO_ATOMICS but
"this is intended for internal development only, to check the
refcounting is properly coded.  It should never become a configuration
option, hence the name of the macro.":
503d4745a1

Atomics are not available in Buildroot if:
 - architecture is 32 bit and something other than ARM or xtensa, and
 - GCC < 4.8 or no threads or FLAT.

The nothreads case can theoretically happen in many different
situations, but in practice nobody disables threads. So the only
interesting case is the FLAT case. Since ARM and RISC-V 64 both have
atomics intrinsics, that leaves just m68k NOMMU as FLAT. So this is
truly a corner case.

The proper solution would be to patch GCC to also provide libatomic in
those cases.
- For nothreads, atomics are in fact not needed, so libatomic can simply
  be implemented as stubs.
- For FLAT, it's probably just a matter of having a match to uclinux in
  libatomic/configure.tgt.

Again, though, this happens only in such niche cases that it's not worth
working on it.

Fixes:
 - http://autobuild.buildroot.org/results/bce526d538f43a541fdfbc0c9b4a7cecebbbc539

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-09-18 15:36:36 +02:00
Luca Ceresoli
ca9fbfd5ee utils/test-pkg: show default build dir in help test
Avoid the "Where did it put my builds?" step.

Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-09-17 17:57:34 +02:00
Fabrice Fontaine
485aa2d691 utils/genrandconfig: add mxs-bootlets custom handling
Add custom cases to make sure that a random configuration with an empty
git or tarball location for mxs-bootlets doesn't fail. It reverts to
BR2_TARGET_MXS_BOOTLETS_FREESCALE in that case.

Fixes:
 - http://autobuild.buildroot.org/results/dcbeb73d152c79c18b1fa3bef3b4fa07635d7b36
 - http://autobuild.buildroot.org/results/f61cf3fc58db9ded5ec42ebf4a9847584700698d

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-09-06 21:58:49 +02:00
Arnout Vandecappelle
1fc299ad6a utils/genrandconfig: remove spurious semicolon
From flake8:
utils/genrandconfig:429:21: E703 statement ends with a semicolon
1     E703 statement ends with a semicolon

Fixes: d3e029575c

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-08-24 13:21:39 +02:00
Fabrice Fontaine
fee46b54e7 utils/genrandconfig: add aufs-util handling
Add a custom case to make sure that a random configuration with an empty
version for aufs-util doesn't fail.

Fixes:
 - http://autobuild.buildroot.org/results/e242cf66a02983bcf6e95b37f8e458bd18aee683

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-24 13:16:27 +02:00
Fabrice Fontaine
d3e029575c utils/genrandconfig: improve ARM trusted firmware fixup
Improve commit 541e794a95 by adding a
custom case to make sure that a random configuration with an empty
platform for arm-trusted-firmware doesn't fail:

make_helpers/plat_helpers.mk:15: *** "Error: Unknown platform. Please use PLAT=<platform name> to specify the platform".  Stop.

Fixes:
 - http://autobuild.buildroot.org/results/1b67220008223d1bcbe70b76d643f9d04362ba6b

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 23:55:49 +02:00
Yann E. MORIN
b6bfa3f744 utils/genrandconfig: dump traceback for unhandled exceptions
In case of an unexpected error, we currently only print the exception as
an str(). For example, the recent issue with the glibc version check
only reported:
    TypeError: cannot use a string pattern on a bytes-like object

That does not help in fixing the issue; the exception text is also not
usually very user-friendly either anyway.

We change the reporting to print the traceback, which in the glibc
version check mentioned above, the error is reported as:

    Traceback (most recent call last):
      File "./utils/genrandconfig", line 740, in <module>
        ret = gen_config(args)
      File "./utils/genrandconfig", line 676, in gen_config
        if not is_toolchain_usable(configfile, toolchainconfig):
      File "./utils/genrandconfig", line 186, in is_toolchain_usable
        if StrictVersion('2.14') > StrictVersion(glibc_version):
      File "/usr/lib/python3.8/distutils/version.py", line 40, in __init__
        self.parse(vstring)
      File "/usr/lib/python3.8/distutils/version.py", line 135, in parse
        match = self.version_re.match(vstring)
    TypeError: cannot use a string pattern on a bytes-like object

With this, the error is much easier to pinpoint (it's the last one that
is not in a system module).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 22:19:52 +02:00
Yann E. MORIN
12e4f7c5c4 utils/genrandconfig: fix checking host glibc version
Unless explicitly told otherwise, subprocess.check_output() returns
bytes objects [0].

When we try to check the C library version (to check the Linaro
toolchain is usable), genrandconfig currently fails with:
    TypeError: cannot use a string pattern on a bytes-like object

So, as suggested in the python documentation, decocde() the output of
subprocess.check_output() before we can use it.

[0] https://docs.python.org/3/library/subprocess.html#subprocess.check_output

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-23 22:19:45 +02:00
Fabrice Fontaine
2d3d130bc1 utils/genrandconfig: add opensbi custom handling
Add custom cases to make sure that a random configuration with an empty
git, tarball location or version for opensbi doesn't fail. It reverts to
BR2_TARGET_OPENSBI_LATEST_VERSION in that case.

Fixes:
 - http://autobuild.buildroot.org/results/b31882b1a7119b2b590cc6dfed44fe515a63a6b8
 - http://autobuild.buildroot.org/results/61866e8941741c9746dcb2c60b6db3b0e5b9a47b

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-20 11:31:56 +02:00
James Hilliard
69400611b2 utils/scanpypi: restore modules search path in case of error
We extend the modules search path to be able to load the package
metadata. Currently, it is only restored when loading those
succeeded, not when it failed.

Restore it to its previous state also in case of error, to avoid
leaking the path further.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-08-17 21:21:50 +02:00