On some cases, when the package contains multiple license files
and some of them from the same type, the scanpypi script will write
the same license type more than once under _LICENSE.
Hence, before creating the _LICENSE variable, we'll remove every
possible duplication.
Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This commit modifies the main() function so that it returns the sum of
build and legal errors, making sure the overall test-pkg script exists
with a non-zero error code upon failure.
Signed-off-by: Heiko Thiery <heiko.thiery@kontron.com>
[Thomas: improved commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
'source' without a previous 'menu' is common in package/Config.in in
br2-externals.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This makes sure the state from a previous run (previous file) can never
leak over into the next file.
Also order the initializations alphabetically.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The handling of 'comment...', 'if ...' and 'menu ...' lines have almost
nothing in common, and subsequent patches will give them even less in
common. Therefore, completely separate their handling in top-level
conditions. The only code that gets duplicated in the different branches
is the 'self.initialize_level_elements(text)' call.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
A comment is considered an alternative delimiter like a menu. I.e.,
a menu that comes after a comment should not be considered a submenu of
that comment. Therefore, remove the '-comment' state before adding the
'-menu' one.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Change the type of variable 'new_package' to make it a class member.
It will be used not only locally. Also initialize it.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Factor out two functions to initialize arrays elements. They will be
reused by followup patches.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Get value of variable 'level' only just after the state change.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Change the type of variable "level" to make it a class member.
It will be used not only locally.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: initialize self.level in the before() method, as suggested by
Ricardo]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
MANIFEST may contains line like this:
"LICENSE LICENSE file (added by Distar)"
so, retains only the first word.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
After a few weeks of running reproducible builds in the autobuilders,
we found out that such builds without a diffoscope analysis are pretty
useless: the cmp on the tarballs doesn't help us fix the
reproducibility issue.
So, let's only do reproducible builds when diffoscope is available.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In a future patch, we are going to need a SystemInfo() instance in
gen_config(), so create the SystemInfo() instance there, and pass it
to fixup_config(), where we currently use it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fix a weird blank-line missing and 2 trailing blank-space.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch extends the Developer class so that it associates each
developer with the defconfigs (in configs/) is in responsible for,
according to the DEVELOPERS file.
It will allow using the getdeveloperlib module to find which developer
is responsible for which defconfig, and send e-mail notifications of
defconfig build failures.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch extends the Developer class so that it associates each
developer with the runtime tests (in support/testing) is in
responsible for, according to the DEVELOPERS file.
The implementation relies on the unittest module to list all test
cases and does some manual parsing of these test-case objects to get
the actual list of test-cases per test-suite.
A global variable is used to compute the list of unittest only once.
This feature will allow to use the getdeveloperlib module to find
which developer is responsible for which runtime test, and send e-mail
notifications of runtime tests failures.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The external.mk file in a br2-external usually contains raw makefile
targets. This file is common code and not a package recipe so it should
not be tested against the code-style of a package .mk file.
When using this script to check files in a br2-external tree, usually
the user is responsible for not passing files that check-package do not
understand. But external.mk is special because it is part of the
br2-external structure, so it is likely someone expects it to be
checkable by an in-tree script.
Instead of adding another blob to the manual, just ignore this file.
Only do that when a br2-external is being tested (so with option -b
passed to the script) and also check that it is on the root path of the
br2-external to allow someone to have a package called external.
Reported on bug #11271.
Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
One of the possible usages of check-package is to first cd to the
directory that contains the files to test (e.g. a package directory) and
then call the script passing the files in the current dir.
It already works when used for intree files, but for files in a
br2-external it throws an exception because some check functions (from
utils/checkpackagelib/lib_*.py) do need the name of the file being
processed and assume there will be a slash before the name.
Fix all check functions that assume that the full filename being checked
contains a slash. Do not use regexps to extract the filename, use
os.path functions instead.
Notice RemoveDefaultPackageSourceVariable and TypoInPackageVariable lead
to an exception in this case, but ApplyOrder instead generates a false
warning.
Fixes bug #11271.
Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fix an issue introduced by Arnout while committing. Jerzy originally
initialized the menu_of_packages, package and print_package_warning
members like they should be, but Arnout thought it wasn't needed and
removed that.
It is actually needed, to make sure the top level (level 0) works.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/264383157
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The 'source' strings identify which package is incorrectly ordered. We
need to extract the actual package name from that string, which is
currently done with constants that assume the file is package/Config.in.
In addition, only 'source' lines that are indented with a tab are
checked. This kind of indentation is done in package/Config.in, but not
e.g. boot/Config.in.
Therefore, use a regular expression to match the 'source' lines, and to
extract the directory part from it.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the future, the nesting level of menus, comments and conditions may
increase. The fixed array length used now is not appropriate. Therefore,
append elements to the arrays if needed.
Also change order of variables.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The CommentsMenusPackagesOrder check builds the 'state' to track the
depth of menus and conditions. However, a menuconfig doesn't create a
menu by itself - it is always followed by a condition that implies the
menu. As a result, when unwinding the 'state', the level will be wrong.
Fix this by checking for menu followed by a space, so it no longer
matches menuconfig. For consistency, do the same for comment and if
as well.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The CommentsMenusPackagesOrder test is broken in various ways for files
other than package/Config.in and package/Config.in.host. Therefore, the
script gives bogus errors for various other Config.in files.
However, we don't really want to check those other files. Indeed, many
of them have a non-alphabetical ordering for good reasons.
Therefore, skip the check for files other than package/Config.in and
package/Config.in.host.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/251214899
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
[Arnout:
- calculate level by counting - instead of with a static array;
- new_package is only used locally, so don't make it a class member;
- do indentation according to length of prefix;
- don't split string in the middle of a line;
- report first wrong package per menu;
- do replace() only once;
- add comment why we do replace().
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When running in a CI system, stat messages become white noise. Introduce
an option to suppress non-error, non-warning, messages.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
test-pkg will use gigabytes of space when testing all toolchains.
Nevertheless, you are normally only interested in the actual build / host
tree when there is a build failure.
Do a 'make clean' for successful builds to save disk space, unless the new
option '-k/--keep' is set.
Note that the logfile and configuration is always retained for inspection.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The long option parsing of test-pkg is broken because:
- some long options are not declared
- there should be a comma between long options, the colon does not replace
it.
This change also revealed that the declaration of 'toolchains-dir' should
have been 'toolchains-csv', originally introduced in commit ed59f81a3c.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Initial patch to start reproducibility testing.
Also enable tar filesystem to compare images since diffoscope works well
with tar.
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Instead of stdout where it gets mixed with the normal output, confusing
software parsing the output (E.G. get-developers -e as git sendemail.ccCmd).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The XS modules have a high propability to fail (compared to Pure Perl modules),
so it is valuable to check XS dependencies before the check of the main module.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
It is possible to generate one-line config for the package just by
normalize it to the form:
BR2_PACKAGE_${pkg_replaced-to_and_uppercase}
it simplifes a bit of testing package where no additional config options
are needed.
Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some python packages seem to use underscores in inconsistent ways. We can
attempt to normalize these by always using dashes for the buildroot name and
attempting to autodetect the correct metadata name format.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
For details, see https://github.com/snyk/zip-slip-vulnerability
Older python versions do not validate that the extracted files are inside
the target directory. Detect and error out on evil paths before extracting
.zip / .tar file.
Given the scope of this (zip issue was fixed in python 2.7.4, released
2013-04-06, scanpypi is only used by a developer when adding a new python
package), the security impact is fairly minimal, but it is good to get it
fixed anyway.
Reported-by: Bas van Schaik <security-reports@semmle.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that Qt4 is removed, these fixups can be removed as well.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
For the general case, appending values to variables is OK and also a
good practice, like this:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR += value2
or this, when the above is not possible:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR := $(PACKAGE_VAR), value2
But this override is an error:
|PACKAGE_VAR = value1
|PACKAGE_VAR = value2
as well this one:
|ifeq ...
|PACKAGE_VAR += value1
|endif
|PACKAGE_VAR = value2
And this override is error-prone:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR = value2
Create a check function to warn about overridden variables.
Some variables are likely to have a default value that gets overridden
in a conditional, so ignore them. The name of such variables end in
_ARCH, _CPU, _SITE, _SOURCE or _VERSION.
After ignoring these variable names, there are a few exceptions to this
rule in the tree. For them use the comment that disables the check.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Simon Dawson <spdawson@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently check-package only knows about ifeq/ifneq.
Add code to handle ifdef/ifndef as well.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently any exceptions for a check function need to be coded into the
check-package script itself.
Create a pattern that can be used in a comment to make check-package
ignore one or more warning types in the line immediately below:
# check-package Indent, VariableWithBraces
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This script currently uses "/usr/bin/env python" as shebang but it does
not really support Python3. Instead of limiting the script to Python2,
fix it to support both versions.
So change all imports to absolute imports because Python3 follows PEP328
and dropped implicit relative imports.
In order to avoid errors when decoding files with the default 'utf-8'
codec, use errors="surrogateescape" when opening files, the docs for
open() states: "This is useful for processing files in an unknown
encoding.". This argument is not compatible with Python2 open() so
import 'six' to use it only when running in Python3.
As a consequence the file handler becomes explicit, so use it to close()
the file after it got processed.
This "surrogateescape" is a simple alternative to the complete solution
of opening files with "rb" and changing all functions in the lib*.py
files to use bytes objects instead of strings. The only case we can have
non-ascii/non-utf-8 files being checked by the script are for patch
files when the upstream file to be patched is not ascii or utf-8. There
is currently one case in the tree:
package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 62d5558f76 (utils/get-developers: make it callable from elsewhere
than the toplevel directory) tried to fix this by passing in the toplevel
directory when the DEVELOPERS file is parsed.
Unfortunately this is not enough, as E.G. also the paths listed in the
patches are relative to the toplevel directory, causing it to not match the
entries in the DEVELOPERS file.
In concept this can be fixed by also passing the toplevel directory to the
Developers class, but the simplest solution is just to chdir to the toplevel
Buildroot directory before calling any of the getdeveloperlib functions.
This does require us to finish parsing command line arguments (which opens
the provided patch files) to not get into trouble with relative paths to
patches before chdir'ing / initializing getdeveloperlib.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
