Commit Graph

265 Commits

Author SHA1 Message Date
Thomas Petazzoni
0e64537917 utils/genrandconfig: reduce the maximum "size" of random configurations
genrandconfig is used by the Buildroot autobuilders to generate
semi-random configurations that we build test. As part of this, we use
"make randpackageconfig" to randomize the selection of packages,
together with a KCONFIG_PROBABILITY value, which indicates the
probabibility for each option to be enabled. This probability is
itself randomized, between 1% and 30% for every build.

However, with our increasing number of packages (over 2900), when we
use a 30% probability for options to be enabled, it means a *lot* of
options are enabled, causing very large configurations to be
tested. These configurations are not very realistic, and they take
ages to build on our autobuilders: we have builds that take 4, 5 or
even 7 hours to build.

In order to test a larger number of configurations and therefore a
larger variety of configurations, this commit reduces the maximum
probability to 20%.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-11-14 13:50:50 +01:00
Thomas Petazzoni
79cba4056b utils/getdeveloperlib.py: call Developer.hasfile() with relative path
In commit
40bb37bd70 ("utils/getdeveloperlib.py:
use relative paths for files"), the Developer class was changed to use
relative paths, including for its .hasfile() method.

However the check_developers() function of getdeveloperlib.py was not
updated accordingly, and continued to pass absolute paths. This caused
"get-developers -c" to return the entire list of files in Buildroot as
being unmaintained, as none of them were matching the file listed in
the DEVELOPERS file.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-02 23:20:11 +01:00
Thomas Petazzoni
53da6a7c05 utils/getdeveloperlib.py: fix check_output() return value decoding
In Python 3.x, check_output() returns a "bytes" array, and not a
string. Its result needs to be decoded to be turned into a
string. Without this fix, "get-developers -c" bails out with:

Traceback (most recent call last):
  File "/home/thomas/projets/buildroot/./utils/get-developers", line 105, in <module>
    __main__()
  File "/home/thomas/projets/buildroot/./utils/get-developers", line 53, in __main__
    files = getdeveloperlib.check_developers(devs)
  File "/home/thomas/projets/buildroot/utils/getdeveloperlib.py", line 280, in check_developers
    files = subprocess.check_output(cmd).strip().split("\n")
TypeError: a bytes-like object is required, not 'str'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-02 23:20:06 +01:00
Bernd Kuhls
8d469858ce utils/genrandconfig: test configurations with BR2_PACKAGE_PYTHON3_PY_ONLY
Python3 variant of
https://git.buildroot.net/buildroot-test/commit/?id=c0de21d9530af53eae5588d99d90c7e0cb87c543

to support Kodi 19 which depends on python3:
https://git.buildroot.net/buildroot/commit/?id=148e695e37561fe45d4726cb68f6454464d17797

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-10-18 22:10:08 +02:00
Petr Vorel
d50290764e utils/scanpypi: remove python2 compatibility imports
ee8b680816 ("utils/scanpypi: use python3 explicitly") started to use python3,
thus compatibility can be removed:

from __future__ import print_function
from __future__ import absolute_import

Tested with python3 -m py_compile.

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-22 21:40:43 +02:00
Petr Vorel
35f15db30a support: utils: use python3 explicitly
Python 2 is EOL sice 2020 [1], it's still available on distros, but may not
be installed by default (as being replaced by python3).

Thus remove compatibility imports:
from __future__ import print_function
from __future__ import absolute_import

Tested with python3 -m py_compile.

[1] https://www.python.org/doc/sunset-python-2/

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-22 21:30:24 +02:00
James Knight
eb75d71b80 utils/getdeveloperlib.py: force foward-slash for pattern matching
Force relative file path resolution of DEVELOPERS file entries to use
forward-slash separators since pattern matching assumes forward slashes.
This is to help permit uses invoking `get-developers` on Platforms where
`os.sep` may not be a forward slash.

Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-11 22:33:42 +02:00
James Knight
9f127cc420 utils/getdeveloperlib.py: explicitly set devs document encoding
Explicitly indicate the file encoding to UTF-8 for the DEVELOPERS
document. This prevents Unicode decoding errors when printing E-Mail
entries with Unicode characters on systems using an alternative default
encoding (e.g. 'CP1252').

This corrects the following observed error:

    $ ./utils/get-developers outgoing/*
    Traceback (most recent call last):
      File "utils\get-developers", line 105, in <module>
        __main__()
      File "utils\get-developers", line 47, in __main__
        devs = getdeveloperlib.parse_developers()
      File "...\buildroot\utils\getdeveloperlib.py", line 239, in parse_developers
        for line in f:
      File "...\Python<ver>\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6659: character maps to <undefined>

Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-09-05 16:08:10 +02:00
Yann E. MORIN
a30204b005 utils/test-pkg: add mode to only prepare .config files
Currently, running test-pkg is only done locally on the developers
machine.

In a follow up commit, we'll add the possibility to run test-pkg in a
gitlab-ci pipeline and, to speed things up, with one job per buildable
configuration.

As such, we will need to make sure that test-pkg only prepares the
configurations, and that it does not build them.

Add such a mode, with a new option, --prepare-only

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-28 17:01:11 +02:00
Yann E. MORIN
3418a068be utils/test-pkg: remove configurations that are skipped
When the config fragment provided by the user is not usable with a
specific toolchain configuration, the resulting .config file was kept
around.

In a follow up commit, we'll need to know, from outside test-pkg, if a
specific configuration was indeed usable or not.

So, unless if the user actually requested to keep the build directories,
remove the .config file when it contains a configration that would be
skipped.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-05 22:35:24 +02:00
Arnout Vandecappelle (Essensium/Mind)
f982f70434 utils/scanpypi: fix flake8 errors
Commit e43c050944 introduced two flake8
errors:

utils/scanpypi:300:26: E231 missing whitespace after ','
utils/scanpypi:302:9: F841 local variable 'setup' is assigned to but never used

The first one is easily fixed. The second one needs a little bit of
explanation. Before commit e43c0509, the return value of
imp.load_module() was used to be able to explicitly call the 'setup'
function in it in case the metadata was not populated. Since that
commit, calling that function is no longer needed, since setup.py is
executed in exactly the same way as when it's run from the command line,
so if that doesn't work, it's completely broken anyway. Therefore, we
can simply discard the return value of imp.load_module().

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-07-27 23:29:07 +02:00
Thomas De Schampheleire
e43c050944 utils/scanpypi: allow installation of commands without 'main' method
In case the setup.py file of a python package does not directly call the
'setup' method, utils/scanpypi was hoping there be a 'main' function which
would do the work, normally called via a construct like:

    if __name__ == '__main__':
        main()

However, this construct is nonstandard, and there are packages in PyPI which
call 'setup()' directly from the 'if' statement, without a main() method.

But scanpypi does not actually need to make such assumption: when loading
the module, it can decide the name to be '__main__', just as if setup.py
would be loaded interactively.

Additionally, remove some logic seemingly related to the previous trick of
calling 'main'. There should not be a problem in keeping already loaded
modules in sys.modules, as this is the purpose of sys.modules.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-07-25 23:43:13 +02:00
Thomas De Schampheleire
ad042904f4 utils/scanpypi: add setup.py script directory as sys.path[0]
Even though the directory containing a package's setup.py was added to
sys.path, some setup.py implementations rely on the fact that it is placed
in sys.path[0].

An example package is 'cram' which failed to be added with scanpypi:

    Traceback (most recent call last):
      File "utils/scanpypi", line 756, in <module>
        main()
      File "utils/scanpypi", line 703, in main
        package.load_setup()
      File "utils/scanpypi", line 303, in load_setup
        setup = imp.load_module('setup', s_file, s_path, s_desc)
      File "/usr/lib/python3.8/imp.py", line 234, in load_module
        return load_source(name, filename, file)
      File "/usr/lib/python3.8/imp.py", line 171, in load_source
        module = _load(spec)
      File "<frozen importlib._bootstrap>", line 702, in _load
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 783, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 44, in <module>
        long_description=long_description(),
      File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 20, in long_description
        return open(os.path.join(sys.path[0], 'README.rst')).read()
    FileNotFoundError: [Errno 2] No such file or directory: '.../buildroot/utils/README.rst'

The corresponding code from cram's setup.py is:

    def long_description():
        """Get the long description from the README"""
        return open(os.path.join(sys.path[0], 'README.rst')).read()

Indeed, the Python documentation says:

https://docs.python.org/3.8/library/sys.html#sys.path
    "...
    As initialized upon program startup, the first item of this list,
    path[0], is the directory containing the script that was used to invoke
    the Python interpreter.
    ..."

Fix this by inserting explicitly at index 0 instead of appending to
sys.path.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-07-25 23:43:11 +02:00
Fabrice Fontaine
60aa896904 utils/scanpypi: search LICENSE.MD
Some packages such as python-idna has a LICENSE.md file:
https://github.com/kjd/idna/blob/master/LICENSE.md

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-07-04 22:04:28 +02:00
Peter Korsgaard
8d07baab43 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-06-07 17:14:37 +02:00
Thomas De Schampheleire
7c9dfe4fd9 utils/genrandconfig: also test BR2_ENABLE_RUNTIME_DEBUG
... similar to BR2_ENABLE_DEBUG.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-06-01 22:40:12 +02:00
Matthew Weber
4e55bc8a4d utils/genrandconfig: drop hardening Config enables
Since 810ba387be, some form of these options are enable
by default. Specifically:

- Kept FORTIFY level 2 option as the default is now level 1.
- Removed all SSP options as the default now uses the best
  option based on toolchain support.
- Similar to SSP, for RELRO, the default now uses the best
  option based on toolchain support.
- Completely drop PIC PIE as it defaults =y

Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-24 15:29:26 +02:00
Thomas Petazzoni
ee8b680816 utils/scanpypi: use python3 explicitly
scanpypi is python3 compatible. In addition, it executes the setup.py
of Python modules to extract the relevant information. Since these are
more and more commonly using python3 constructs, using "python" to run
scanpypi causes problems on systems that have python2 installed as
python, when trying to parse setup.py scripts with python3 constructs.

Fixes part of #13516.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-20 17:42:46 +01:00
Thomas Petazzoni
38b0560f4e utils/getdeveloperlib.py: reduce Cc: list based on package infras
When a developer has package/pkg-<infra>.mk assigned to him/her in the
DEVELOPERS file, this has 3 implications:

 (1) Patches adding new packages using this infrastructure are Cc'ed
     to this developer. This is done by the analyze_patch() function,
     which matches the regexp r"^\+\$\(eval
     \$\((host-)?([^-]*)-package\)\)$" in the patch, i.e where an
     added line contains a reference to the infra maintained by the
     developer.

 (2) Patches touching the package/pkg-<infra>.mk file itself are Cc'ed
     to this developer.

 (3) Any patch touching a package using this infra are also Cc'ed to
     this developer.

Point (3) causes a significant amount of patches to be sent to
developers who have package/pkg-generic.mk and
package/pkg-autotools.mk assigned to them in the DEVELOPERS
file. Basically, all patches touching generic or autotools packages
get CC'ed to such developers, which causes a massive amount of patches
to be received.

So this patch adjusts the getdeveloperlib.py to drop point (3), but
preserves point (1) and (2). Indeed, it makes sense to be Cc'ed on new
package additions (to make a review that they use the package
infrastructure correctly), and it makes sense to be Cc'ed on patches
that touch the infrastructure code itself.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-10 22:10:58 +01:00
Francois Perrad
9e57d626c3 package/perl: bump to version 5.32.1
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-25 22:29:32 +01:00
Yann E. MORIN
9277978e28 utils: fix flake8 warning
Commit 40bb37bd70 refactored get-developers, and now the 'os' module is
no longer needed, but still imported:

    utils/get-developers:6:1: F401 'os' imported but unused
    1     F401 'os' imported but unused

Drop it now.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-01-23 21:24:48 +01:00
Thomas Petazzoni
40bb37bd70 utils/getdeveloperlib.py: use relative paths for files
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.

parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.

This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.

Before this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]

$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": []
}

After this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
  "Floris Bos <bos@je-eigen-domein.nl>",
  "Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": [
    "Christophe Priouzeau <christophe.priouzeau@st.com>"
  ]
}

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:08 +01:00
Thomas Petazzoni
57ecb6c8eb utils/get-developers: use Developers.hasfile() where appropriate
Instead of open-coding Developers.hasfile() in utils/get-developers,
use it directly.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:07 +01:00
Thomas De Schampheleire
a1bb132a81 utils/checkpackagelib/lib_mk.py: handle 'else' and 'elif' statements
An 'else' or 'elif' clause inside a make conditional should not be indented
in the same way as the if/endif clause. check-package did not recognize the
else statement and expected an indentation.

For example:

ifdef FOOBAR
	interesting
else
	more interesting
endif

would, according to check-package, need to become:

ifdef FOOBAR
	interesting
	else
	more interesting
endif

Treat 'else' and 'elif' the same as if-like keywords in the Indent test, but
take into account that 'else' is also valid shell, so we need to correctly
handle line continuation to prevent complaining about the 'else' in:

ifdef FOOBAR
	if true; \
	    ... \
	else \
	    ... \
	fi
endif

We don't add the 'else' and 'elif' statements to start_conditional, because
it would cause incorrect nesting counting in class OverriddenVariable.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-02 13:54:59 +01:00
Francois Perrad
b689a5a720 utils/scancpan: add GPL* as license file
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-29 23:00:53 +01:00
Francois Perrad
1120980f30 utils/scancpan: handles README.pod as license file
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-29 23:00:45 +01:00
Heiko Thiery
afc112b0e4 utils/getdeveloperlib.py: fix issue with hasfile()
pkg-stats is not able anymore to set the developers for defconfigs and
packages. This issue is introduced with
ae86067a15. The hasfile() method from
Developer object tries to check an absolute path against a relative path.

Convert the filepath to be checked also into an absolute path.

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-17 23:47:01 +01:00
Matt Weber
3930fd2ddd genrandconfig: uboot-tools env/scr creation test files
Normally the kconfig stings would end up empty and cause a build
error.  This patch provides test files to allow testing the creation
of uboot environment and script bin files from user provided txt files.

Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2020-10-06 23:37:48 +02:00
Thomas Petazzoni
f9150a6a3d utils/scanpypi: use raw strings in re.compile/re.sub
Fixes the following Python 3.x flake8 warning:

W605 invalid escape sequence '\w'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:56:17 +02:00
Thomas Petazzoni
c5b848d719 utils/getdeveloperlib.py: use raw strings for re.compile/re.match
Fixes the following Python 3.x flake8 warning:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:56:11 +02:00
Thomas Petazzoni
163f160a8e utils/{check-package, checkpackagelib}: consistently use raw strings for re.compile
Raw strings need to be used when calling re.compile() otherwise Python
3.x flake8 complains with:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:55:59 +02:00
Fabrice Fontaine
e15c35f588 utils/scancpan: use two spaces indentation in hash file
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-15 23:17:46 +01:00
Peter Korsgaard
fd99eb5016 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-09 15:17:09 +01:00
Peter Korsgaard
06417e97e3 utils/genrandconfig: drop outdated python-nfc check
Commit 9ea528f84b (package/python-nfc: bump to version 0.13.5) changed the
python-nfc package to download from github, so the package no longer needs
bzr on the host.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-04 19:08:12 +01:00
Peter Korsgaard
22e833af5e Config.in: drop BR2_NEEDS_HOST_{JAVAC,JAR}
With classpath removed, no packages select these symbols any more - So drop
them and their corresponding logic in dependencies.sh / genrandconfig.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-03 23:55:48 +01:00
James Hilliard
9fbbf4fd2a utils/scanpypi: update hash file indentation formatting
The new .hash convention is to use 2 spaces between fields.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-26 20:53:33 +01:00
Yann E. MORIN
76772cefd8 utils/check-package: ignore ACLOCAL_PATH
The aclocal program is provided by the automake package, so it makes
sense to define aclocal-related variables in automake.mk.

Add an exception to check-package to ignore that variable.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-04 17:15:19 +01:00
Francois Perrad
5bff3945af utils/scancpan: warn when a module is a perl core module
we don't want create new BR package with perl core module,
because core modules are already included in perl distribution,
and built with the BR package perl.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-08 18:16:54 +01:00
Matt Weber
f64701b03d utils/scanpypi: remind developer about updating DEVELOPERS and Config.in
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-26 09:32:32 +01:00
Thomas Petazzoni
e6bbb78577 utils/genrandconfig: test per-package directories
Now that the support for per-package directories has been merged, it
is time to get some exposure for it in the autobuilders, so let's
build 1 out of 15 builds with this feature enabled, at least as an
initial step.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-05 10:30:48 +01:00
Francois Perrad
49b3a66be6 utils/scancpan: follow Perl version
linked to https://git.busybox.net/buildroot/commit?id=01134ca99a2c83932aee42984c81e51cc0428425

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-03 23:12:38 +01:00
Thomas Petazzoni
5cfe5d7897 utils/genrandconfig: fix runtime issue with Python 3
With Python 3.7, genrandconfig fails with:

'str' object has no attribute 'decode'

We are already working on str objects, and there is no need to decode
them, so we drop the call to decode_byte_list() and its definition as
it was only used there.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-03 23:00:01 +01:00
Bartosz Bilas
2d28d0203a utils/scanpypi: sort alphabetically list of required packages
That change will alphabetically set list order
of required packages in Config.in file automatically.

Example below:

before: ['python-pyserial', 'python-pyaes', 'python-ecdsa']
after:  ['python-ecdsa', 'python-pyaes', 'python-pyserial']

Signed-off-by: Bartosz Bilas <b.bilas@grinn-global.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-08 21:53:02 +01:00
Francois Perrad
f42e355b56 utils/scancpan: handle LICENCE spelling
License files are sometimes called LICENCE, so support that as well.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-29 23:46:46 +01:00
Matt Weber
17eb6ae92a utils/genrandconfig: test full set of hardening options
This patch adds the remaining ssp and relro cases plus both of the
fortify options. The randomization was left consistant between the
options but the order of the conditions placed the most restrictive
hardening options with more priority.

Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 20:20:41 +01:00
Fabrice Fontaine
22d37003c7 utils/genrandconfig: test configurations with BR2_SSP_ALL
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 20:20:37 +01:00
Fabrice Fontaine
d36da5a5f5 utils/genrandconfig: test configurations with BR2_RELRO_FULL
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 20:20:24 +01:00
Yann E. MORIN
1e03a2e89e utils/check-package: report := that appends to variables
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-27 10:35:06 +01:00
Asaf Kahlon
9fc6bfb4b8 utils/genrandconfig: use randint instead of (undefined) randInt
Fixes https://gitlab.com/buildroot.org/buildroot/-/jobs/333788455

Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:25:31 +02:00
Fabrice Fontaine
f24bb4ee4f utils/genrandconfig: test configurations with BR2_PIC_PIE
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 15:02:57 +02:00
Asaf Kahlon
143f135833 scanpypi: write every license file once
On some cases, when the package contains multiple license files
and some of them from the same type, the scanpypi script will write
the same license type more than once under _LICENSE.
Hence, before creating the _LICENSE variable, we'll remove every
possible duplication.

Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-11 23:04:40 +02:00
Heiko Thiery
5093435f66 utils/test-pkg: ensure to exit with an error upon failure
This commit modifies the main() function so that it returns the sum of
build and legal errors, making sure the overall test-pkg script exists
with a non-zero error code upon failure.

Signed-off-by: Heiko Thiery <heiko.thiery@kontron.com>
[Thomas: improved commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-10 22:54:12 +02:00
Jerzy Grzegorek
31fc6a1fe4 utils/checkpackagelib: CommentsMenusPackagesOrder: add more Config.in files to check
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: remove boot/Config.in, it is not ordered correctly yet.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
16271ec356 utils/checkpackagelib: CommentsMenusPackagesOrder: initialize 'menu_of_packages' array
'source' without a previous 'menu' is common in package/Config.in in
br2-externals.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
b47590fdb0 utils/checkpackagelib: CommentsMenusPackagesOrder: initialize in before()
This makes sure the state from a previous run (previous file) can never
leak over into the next file.

Also order the initializations alphabetically.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
6a5f6ea7e2 utils/checkpackagelib: CommentsMenusPackagesOrder: use '-' to describe state
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
0625899e3b utils/checkpackagelib: CommentsMenusPackagesOrder: separate comment/if/menu cases
The handling of 'comment...', 'if ...' and 'menu ...' lines have almost
nothing in common, and subsequent patches will give them even less in
common. Therefore, completely separate their handling in top-level
conditions. The only code that gets duplicated in the different branches
is the 'self.initialize_level_elements(text)' call.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
dd251d68e5 utils/checkpackagelib: CommentsMenusPackagesOrder: remove '-comment' state before the '-menu' one
A comment is considered an alternative delimiter like a menu. I.e.,
a menu that comes after a comment should not be considered a submenu of
that comment. Therefore, remove the '-comment' state before adding the
'-menu' one.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
c82e0092f5 utils/checkpackagelib: CommentsMenusPackagesOrder: change the type of variable 'new_package'
Change the type of variable 'new_package' to make it a class member.
It will be used not only locally. Also initialize it.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
7197d3cb80 utils/checkpackagelib: CommentsMenusPackagesOrder: add functions to initialize arrays elements
Factor out two functions to initialize arrays elements. They will be
reused by followup patches.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
4e88b37274 utils/checkpackagelib: CommentsMenusPackagesOrder: get value of variable 'level'
Get value of variable 'level' only just after the state change.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
6748e42ace utils/checkpackagelib: CommentsMenusPackagesOrder: change the type of variable 'level'
Change the type of variable "level" to make it a class member.
It will be used not only locally.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: initialize self.level in the before() method, as suggested by
Ricardo]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
dd99dc5763 utils/checkpackagelib: CommentsMenusPackagesOrder: rename variable 'm'
Rename variable 'm' for better readability.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:39 +02:00
Francois Perrad
f71f944fe6 utils/scancpan: improve license file detection
MANIFEST may contains line like this:
"LICENSE                     LICENSE file (added by Distar)"
so, retains only the first word.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-17 22:51:01 +02:00
Thomas Petazzoni
7d14a659db utils/genrandconfig: only do reproducible builds with diffoscope
After a few weeks of running reproducible builds in the autobuilders,
we found out that such builds without a diffoscope analysis are pretty
useless: the cmp on the tarballs doesn't help us fix the
reproducibility issue.

So, let's only do reproducible builds when diffoscope is available.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-07 16:34:44 +02:00
Thomas Petazzoni
4693676ec5 utils/genrandconfig: instantiate SystemInfo() earlier
In a future patch, we are going to need a SystemInfo() instance in
gen_config(), so create the SystemInfo() instance there, and pass it
to fixup_config(), where we currently use it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-07 16:34:40 +02:00
Victor Huesca
a0a066a8ff utils/getdeveloperlib.py: fix flake8 warnings
Fix a weird blank-line missing and 2 trailing blank-space.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 22:22:01 +02:00
Victor Huesca
caead54274 utils/getdeveloperlib.py: add defconfig parsing
This patch extends the Developer class so that it associates each
developer with the defconfigs (in configs/) is in responsible for,
according to the DEVELOPERS file.

It will allow using the getdeveloperlib module to find which developer
is responsible for which defconfig, and send e-mail notifications of
defconfig build failures.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 00:02:35 +02:00
Victor Huesca
e635836494 utils/getdeveloperlib.py: add runtime test parsing
This patch extends the Developer class so that it associates each
developer with the runtime tests (in support/testing) is in
responsible for, according to the DEVELOPERS file.

The implementation relies on the unittest module to list all test
cases and does some manual parsing of these test-case objects to get
the actual list of test-cases per test-suite.

A global variable is used to compute the list of unittest only once.

This feature will allow to use the getdeveloperlib module to find
which developer is responsible for which runtime test, and send e-mail
notifications of runtime tests failures.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 00:02:09 +02:00
Victor Huesca
bb5576db9b utils/getdeveloperlib: add some debug information
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 23:39:55 +02:00
Ricardo Martincoski
75d6c2ba96 check-package: ignore external.mk
The external.mk file in a br2-external usually contains raw makefile
targets. This file is common code and not a package recipe so it should
not be tested against the code-style of a package .mk file.

When using this script to check files in a br2-external tree, usually
the user is responsible for not passing files that check-package do not
understand. But external.mk is special because it is part of the
br2-external structure, so it is likely someone expects it to be
checkable by an in-tree script.

Instead of adding another blob to the manual, just ignore this file.
Only do that when a br2-external is being tested (so with option -b
passed to the script) and also check that it is on the root path of the
br2-external to allow someone to have a package called external.

Reported on bug #11271.

Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
1842bb1470 check-package: fix check of file in current dir with -b
One of the possible usages of check-package is to first cd to the
directory that contains the files to test (e.g. a package directory) and
then call the script passing the files in the current dir.
It already works when used for intree files, but for files in a
br2-external it throws an exception because some check functions (from
utils/checkpackagelib/lib_*.py) do need the name of the file being
processed and assume there will be a slash before the name.

Fix all check functions that assume that the full filename being checked
contains a slash. Do not use regexps to extract the filename, use
os.path functions instead.

Notice RemoveDefaultPackageSourceVariable and TypoInPackageVariable lead
to an exception in this case, but ApplyOrder instead generates a false
warning.

Fixes bug #11271.

Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Arnout Vandecappelle (Essensium/Mind)
d5990601ba utils/checkpackagelib: CommentsMenusPackagesOrder: properly initialize levels
Fix an issue introduced by Arnout while committing. Jerzy originally
initialized the menu_of_packages, package and print_package_warning
members like they should be, but Arnout thought it wasn't needed and
removed that.

It is actually needed, to make sure the top level (level 0) works.

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/264383157

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-02 18:43:52 +02:00
Arnout Vandecappelle (Essensium/Mind)
0a4e1fc054 utils/checkpackagelib: CommentsMenusPackagesOrder: use regex for source
The 'source' strings identify which package is incorrectly ordered. We
need to extract the actual package name from that string, which is
currently done with constants that assume the file is package/Config.in.

In addition, only 'source' lines that are indented with a tab are
checked. This kind of indentation is done in package/Config.in, but not
e.g. boot/Config.in.

Therefore, use a regular expression to match the 'source' lines, and to
extract the directory part from it.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Jerzy Grzegorek
83a34f7705 utils/checkpackagelib: CommentsMenusPackagesOrder: append elements to arrays if needed
In the future, the nesting level of menus, comments and conditions may
increase. The fixed array length used now is not appropriate. Therefore,
append elements to the arrays if needed.

Also change order of variables.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Arnout Vandecappelle (Essensium/Mind)
c62a282920 utils/checkpackagelib: CommentsMenusPackagesOrder: fix 'menuconfig' handling
The CommentsMenusPackagesOrder check builds the 'state' to track the
depth of menus and conditions. However, a menuconfig doesn't create a
menu by itself - it is always followed by a condition that implies the
menu. As a result, when unwinding the 'state', the level will be wrong.

Fix this by checking for menu followed by a space, so it no longer
matches menuconfig. For consistency, do the same for comment and if
as well.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Arnout Vandecappelle (Essensium/Mind)
5603406bd1 utils/checkpackagelib: CommentsMenusPackagesOrder: only apply to top-level
The CommentsMenusPackagesOrder test is broken in various ways for files
other than package/Config.in and package/Config.in.host. Therefore, the
script gives bogus errors for various other Config.in files.

However, we don't really want to check those other files. Indeed, many
of them have a non-alphabetical ordering for good reasons.

Therefore, skip the check for files other than package/Config.in and
package/Config.in.host.

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/251214899

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-08-01 09:58:15 +02:00
Jerzy Grzegorek
e36a63cf6b checkpackagelib/lib_config.py: check packages alphabetical order in {Config.in, Config.in.host}
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
[Arnout:
 - calculate level by counting - instead of with a static array;
 - new_package is only used locally, so don't make it a class member;
 - do indentation according to length of prefix;
 - don't split string in the middle of a line;
 - report first wrong package per menu;
 - do replace() only once;
 - add comment why we do replace().
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-07-14 00:21:26 +02:00
Brandon Maier
44af8386f8 utils/check-package: Add a quiet flag
When running in a CI system, stat messages become white noise. Introduce
an option to suppress non-error, non-warning, messages.

Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-07-03 23:34:58 +02:00
Thomas De Schampheleire
72bf48606c utils/test-pkg: clean output dir for successful builds
test-pkg will use gigabytes of space when testing all toolchains.
Nevertheless, you are normally only interested in the actual build / host
tree when there is a build failure.

Do a 'make clean' for successful builds to save disk space, unless the new
option '-k/--keep' is set.
Note that the logfile and configuration is always retained for inspection.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-23 21:39:08 +02:00
Thomas De Schampheleire
989cda12ba utils/test-pkg: fix long option parsing
The long option parsing of test-pkg is broken because:
- some long options are not declared
- there should be a comma between long options, the colon does not replace
it.

This change also revealed that the declaration of 'toolchains-dir' should
have been 'toolchains-csv', originally introduced in commit ed59f81a3c.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-23 21:38:13 +02:00
Francois Perrad
280149ce60 utils/scancpan: fix test file creation on br-external
first, add the creation of the needed subdirectories

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-09 23:37:04 +02:00
Atharva Lele
9355933afd utils/genrandconfing: randomly enable BR2_REPRODUCIBLE 10% of the times
Initial patch to start reproducibility testing.

Also enable tar filesystem to compare images since diffoscope works well
with tar.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-08 22:56:41 +02:00
Peter Seiderer
b9b081d53a utils/check-package: fix flake8 warning
Fix flake 8 warning (introduced with commit [1]):

  utils/checkpackagelib/lib.py:56:1: E302 expected 2 blank lines, found 1

[1] https://git.buildroot.net/buildroot/commit/?id=8e352c32b0beded97a8a5c1e9edc9d618514ee7b

Reported-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-20 10:20:38 +02:00
Peter Seiderer
8e352c32b0 utils/check-package: warn about utf-8 characters in .mk files
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-18 23:34:59 +02:00
Thomas Petazzoni
2e1f98d950 utils/genrandconfig: prevent selecting libffi on nds32
libffi does not have support for the nds32 architecture, so let's
disallow libffi on this CPU architecture for the moment.

Fixes:

  http://autobuild.buildroot.net/results/12803a705586e82fdfb49013da2eb3b9879ccd45/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-17 21:17:42 +02:00
Thomas Petazzoni
472e8808d9 utils/genrandconfig: test configurations with BR2_SYSTEM_ENABLE_NLS
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-14 11:26:58 +02:00
Peter Korsgaard
83f82bd67a utils/getdeveloperlib.py: print warnings/errors to stderr
Instead of stdout where it gets mixed with the normal output, confusing
software parsing the output (E.G. get-developers -e as git sendemail.ccCmd).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-04 22:54:53 +02:00
Francois Perrad
f4a30ae5bd utils/scancpan: add a check of XS dependencies in generated test
The XS modules have a high propability to fail (compared to Pure Perl modules),
so it is valuable to check XS dependencies before the check of the main module.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-31 14:01:36 +02:00
Vadim Kochan
a946813dd5 utils/test-pkg: generate package config if it is not specified
It is possible to generate one-line config for the package just by
normalize it to the form:

    BR2_PACKAGE_${pkg_replaced-to_and_uppercase}

it simplifes a bit of testing package where no additional config options
are needed.

Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 10:03:34 +01:00
James Hilliard
f13b843e71 utils/scanpypi: handle underscores in python packages
Some python packages seem to use underscores in inconsistent ways.  We can
attempt to normalize these by always using dashes for the buildroot name and
attempting to autodetect the correct metadata name format.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 12:37:48 +01:00
Peter Korsgaard
a83e30ad63 utils/scanpypi: protect against zip-slip vulnerability in zip/tar handling
For details, see https://github.com/snyk/zip-slip-vulnerability

Older python versions do not validate that the extracted files are inside
the target directory.  Detect and error out on evil paths before extracting
.zip / .tar file.

Given the scope of this (zip issue was fixed in python 2.7.4, released
2013-04-06, scanpypi is only used by a developer when adding a new python
package), the security impact is fairly minimal, but it is good to get it
fixed anyway.

Reported-by: Bas van Schaik <security-reports@semmle.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-12 21:27:35 +01:00
Peter Korsgaard
a11d61e104 utils/genrandconfig: drop qt4 fixups
Now that Qt4 is removed, these fixups can be removed as well.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 16:30:39 +01:00
Ricardo Martincoski
b03fa5d96f utils/check-package: warn about overridden variables
For the general case, appending values to variables is OK and also a
good practice, like this:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR += value2

or this, when the above is not possible:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR := $(PACKAGE_VAR), value2

But this override is an error:
|PACKAGE_VAR = value1
|PACKAGE_VAR = value2

as well this one:
|ifeq ...
|PACKAGE_VAR += value1
|endif
|PACKAGE_VAR = value2

And this override is error-prone:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR = value2

Create a check function to warn about overridden variables.

Some variables are likely to have a default value that gets overridden
in a conditional, so ignore them. The name of such variables end in
_ARCH, _CPU, _SITE, _SOURCE or _VERSION.

After ignoring these variable names, there are a few exceptions to this
rule in the tree. For them use the comment that disables the check.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Simon Dawson <spdawson@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 20:24:57 +01:00
Ricardo Martincoski
fe7a5d7120 utils/check-package: handle ifdef/ifndef in .mk files
Currently check-package only knows about ifeq/ifneq.
Add code to handle ifdef/ifndef as well.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-29 16:38:41 +01:00
Ricardo Martincoski
4a6c5ab2c3 utils/check-package: allow to disable warning for a line
Currently any exceptions for a check function need to be coded into the
check-package script itself.

Create a pattern that can be used in a comment to make check-package
ignore one or more warning types in the line immediately below:
 # check-package Indent, VariableWithBraces

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-29 16:37:47 +01:00
Ricardo Martincoski
02b165dc71 check-package: fix Python3 support
This script currently uses "/usr/bin/env python" as shebang but it does
not really support Python3. Instead of limiting the script to Python2,
fix it to support both versions.

So change all imports to absolute imports because Python3 follows PEP328
and dropped implicit relative imports.

In order to avoid errors when decoding files with the default 'utf-8'
codec, use errors="surrogateescape" when opening files, the docs for
open() states: "This is useful for processing files in an unknown
encoding.". This argument is not compatible with Python2 open() so
import 'six' to use it only when running in Python3.
As a consequence the file handler becomes explicit, so use it to close()
the file after it got processed.

This "surrogateescape" is a simple alternative to the complete solution
of opening files with "rb" and changing all functions in the lib*.py
files to use bytes objects instead of strings. The only case we can have
non-ascii/non-utf-8 files being checked by the script are for patch
files when the upstream file to be patched is not ascii or utf-8. There
is currently one case in the tree:
package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-16 23:14:25 +01:00
Peter Korsgaard
45aabcddc5 utils/get-developers: really make it callable from elsewhere than the toplevel directory
Commit 62d5558f76 (utils/get-developers: make it callable from elsewhere
than the toplevel directory) tried to fix this by passing in the toplevel
directory when the DEVELOPERS file is parsed.

Unfortunately this is not enough, as E.G.  also the paths listed in the
patches are relative to the toplevel directory, causing it to not match the
entries in the DEVELOPERS file.

In concept this can be fixed by also passing the toplevel directory to the
Developers class, but the simplest solution is just to chdir to the toplevel
Buildroot directory before calling any of the getdeveloperlib functions.

This does require us to finish parsing command line arguments (which opens
the provided patch files) to not get into trouble with relative paths to
patches before chdir'ing / initializing getdeveloperlib.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-04 13:23:41 +01:00
Peter Korsgaard
74fbc20504 Revert "utils/get-developers: make it callable from elsewhere than the toplevel directory"
This reverts commit 62d5558f76.

This actually does not work, as patches contain paths relative to the
toplevel directory as well.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-04 13:23:33 +01:00
Adam Duskett
bbf32a77ec utils/test-pkg: force checking dependencies
Currently, if a user runs "make" while specifying a specific package
(IE: make -p foo),  the Makefile logic skips checking to see if all the
dependencies are selected in the specified packages config file. This behavior
is useful to test simple packages which do not have "complex" dependencies.

However; if a developer uses test-pkg -p ${package_name} to check their package,
the package may pass all the checks, but would have otherwise failed with a
simple "make" because the developer may have failed to add a select line in
packages config file, even if there is a new dependency in the packages
Makefile.

Pass the environment variable "BR_FORCE_CHECK_DEPENDENCIES"  to the Makefile in
the test-pkg script,  and check it's value in the Makefile. If the value is
"YES" force checking for dependency issues.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-03 11:41:26 +01:00