Commit Graph

227 Commits

Author SHA1 Message Date
Arnout Vandecappelle (Essensium/Mind)
f1768855e7 utils/size-stats-compare: fix flake8 error
utils/size-stats-compare:80:15: E225 missing whitespace around operator

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-01 21:52:35 +01:00
Thomas De Schampheleire
391f70b689 utils/size-stats-compare: add package name in detail output
size-stats-compare gives an overview of the size increase/decrease between
two cases, based on packages-file-list.txt. The 'detail' mode gives info per
file, otherwise per package.

But sometimes, you want the detailed per-file info, but only for a specific
package. Since the detailed output no longer lists the package name, you
cannot simply grep for it. A workaround was to filter the input
packages-file-list.txt's first, and then pass these filtered versions to
size-stats-compare.

Make this easier by adding the package name next to the filename in detailed
output. This allows grep'ing normally.
For example:

  $ utils/size-stats-compare orig new  -t 100 -d | grep ebtables

      -67712 removed ebtables             lib/ebtables/libebtc.so
      -66764 removed ebtables             lib/ebtables/libebt_nat.so
      -66752 removed ebtables             sbin/ebtables
      -66704 removed ebtables             lib/ebtables/libebt_arp.so
      -66700 removed ebtables             lib/ebtables/libebt_stp.so
      -66700 removed ebtables             lib/ebtables/libebt_among.so
      -66684 removed ebtables             lib/ebtables/libebt_ip.so
      -66676 removed ebtables             lib/ebtables/libebt_limit.so
      -66656 removed ebtables             lib/ebtables/libebt_log.so
      -66648 removed ebtables             lib/ebtables/libebt_mark.so
      -66636 removed ebtables             lib/ebtables/libebt_pkttype.so
      -66604 removed ebtables             lib/ebtables/libebt_vlan.so
      -66588 removed ebtables             lib/ebtables/libebt_ulog.so
      -66588 removed ebtables             lib/ebtables/libebt_nflog.so
      -66584 removed ebtables             lib/ebtables/libebt_arpreply.so
      -66544 removed ebtables             lib/ebtables/libebt_ip6.so
      -66540 removed ebtables             lib/ebtables/libebt_802_3.so
      -66536 removed ebtables             lib/ebtables/libebt_standard.so
      -66524 removed ebtables             lib/ebtables/libebt_mark_m.so
      -66524 removed ebtables             lib/ebtables/libebt_redirect.so
      -66452 removed ebtables             lib/ebtables/libebtable_filter.so
      -66452 removed ebtables             lib/ebtables/libebtable_broute.so
      -66452 removed ebtables             lib/ebtables/libebtable_nat.so
          45         ebtables             etc/ethertypes
       66752 added   ebtables             usr/sbin/ebtablesd
       66752 added   ebtables             usr/sbin/ebtables-legacy
       66752 added   ebtables             usr/sbin/ebtablesu
      200840 added   ebtables             usr/lib/libebtc.so.0.0.0

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
2022-01-31 21:32:22 +01:00
Thomas De Schampheleire
e9b1dffa5d utils/size-stats-compare: clarify meaning of variables in print_result
print_result is juggling with entry[x][y] which is not very readable.
While a better solution would be to use a class and reference named
attributes, that would require some bigger changes in the script.

Instead, make a minimal improvement by assigning the entry[x][y] values to
intermediate variables. Store them in a dict for easy usage from a format
string.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-31 21:24:40 +01:00
Yann E. MORIN
5da8218184 support/download: drop support for the 'none' hash
Commit 1ba85b7f87 (support/download: add explicit no-hash support)
introduced the 'none' hash type, in an attempt to make hash files
mandatory, but not failing on archives localy generated, like those
for git or svn repositories, especially for those packages where a
version choice was present, which would allow for either remote
archives for which we'd have a hash or VCS trees for which we could
not have a hash for the localy generated archive.

Indeed, back in the time, we did not have a mean to generate
reproducible archives, so having a hash file without a hash for
thosel ocally generated archives would trigger an error in the
hash-checking machinery.

But now, low-and-behold, we do know how to generate those archives,
and we have a mechanism to explicitly exclude some archives from being
hash-checked (e.g. when the version string itself can be user-provided).

As such, the 'none' hash type no longer has any raison d'être, we do not
use it in-tree, and its use in a br2-external tree is most probably
inexistent (as is the use of hash files alotgether most probably).

So we simply drop the support for that.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Thomas: drop support in checkpackagelib, as reported by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-01-11 21:58:01 +01:00
Yann E. MORIN
f29239cf9b utils/test-pkg: add check that show-info is proper json
JSON parser are usually rather strict when parsing their input. Both
jq and the python json module choke on non-conformant inputs.

Commit e4c284e6b9 (package/pkg-utils: escape \ in generated legal-info)
fixed the \-escaping case, but we want to ensure that we do not have
other data that would be improperly encoded.

To more easily catch issues, also run show-info and send its output
through jq, to try and validate the output.

Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-10 08:52:58 +01:00
Heiko Thiery
f35a4b4ae2 utils/check-package: add a check for the new spacing convention
The seperation of the fields in the hash file should be 2 spaces for
consistency.

Since a large number of hash files still violate this rule, exclude it
from "make check-package" (and thus from CI).

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout:
 - Move it to a separate class, so it can be excluded.
 - Exclude it from "make check-package"
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-09 23:01:53 +01:00
Arnout Vandecappelle (Essensium/Mind)
50b8d6307f utils/diffconfig: use python3 explicitly
Python 2 is EOL sice 2020 [1], it's still available on distros, but may
not be installed by default (as being replaced by python3). A
compatibility symlink python -> python3 may not be installed either.

Convert the shebang line to the usual /usr/bin/env based line.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-12-29 10:07:59 +01:00
Ricardo Martincoski
29bb478a49 utils/getdeveloperlib.py: fix developer being reported for unrelated path
Currently, by following the instructions in the manual and querying for
developers for a patch that changes path
package/foobar
the script reports both developers that have these entries in the
DEVELOPERS file:
F:	package/foo/
F:	package/foobar/

Starting from commit "afc112b0e4 utils/getdeveloperlib.py: fix issue
with hasfile()" get-developers script uses os.path.abspath() and
os.path.relpath().
The catch is that those functions return the absolute path and the
relative path without the trailing slash.

When the paths associated to a developer are then compared to the paths
a patch touches, using the string.startswith(), any substring returns
True, leading to developers for package/foo/ being wrongly reported
for package/foobar/ .

Fix this by re-adding the trailing slash after using relpath().

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: James Knight <james.d.knight@live.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-11 21:00:29 +01:00
Ricardo Martincoski
5bbedea9c2 utils/checkpackagelib/lib_mk.py: fix check for overridden variable
Currently this .mk snippet results in unexpected behavior from
check-package:
|VAR_1 = VALUE1
|ifeq (condition)
|VAR_1 := $(VAR_1), VALUE2
|endif

Fix commit "163f160a8e utils/{check-package, checkpackagelib}:
consistently use raw strings for re.compile" that ended up doing this:
-    CONCATENATING = re.compile("^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")
+    CONCATENATING = re.compile(r"^([A-Z0-9_]+)\s*(\+|:|)=\s*\$\(\\1\)")

But raw strings do not expect escaping when referencing \1 and the
pattern ends up searching for a raw '\\1' instead of an occurrence of
the first pattern inside parenthesis.

|$ python3
|Python 3.8.10 (default, Sep 28 2021, 16:10:42)
|[GCC 9.3.0] on linux
|Type "help", "copyright", "credits" or "license" for more information.
|>>> import re
|>>> p1 = re.compile('(foo)bar\\1')
|>>> p2 = re.compile(r'(foo)bar\\1')
|>>> p3 = re.compile(r'(foo)bar\1')
|>>> s1 = 'foobarfoo'
|>>> s2 = 'foobar\\1'
|>>> print(p1.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p2.search(s1))
|None
|>>> print(p3.search(s1))
|<re.Match object; span=(0, 9), match='foobarfoo'>
|>>> print(p1.search(s2))
|None
|>>> print(p2.search(s2))
|<re.Match object; span=(0, 8), match='foobar\\1'>
|>>> print(p3.search(s2))
|None
|>>>

So use '\1' instead of '\\1' in the raw string.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-10 20:01:40 +01:00
Fabrice Fontaine
4910a175b3 utils/checkpackagelib/lib_mk.py: check DEPENDENCIES
Check that {FOO}_DEPENDENCIES are never overriden in a conditional

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-10 19:26:30 +01:00
Fabrice Fontaine
fba568a16e utils/checkpackagelib/lib_mk.py: check typo in define
Check typo in define to detect SMAKE_LINUX_CONFIG_FIXUPS in smack
(fixed by 41e2132fbe)

The new expression will catch "SMAKE_CONF_OPTS" as well as
"define SMAKE_LINUX_CONFIG_FIXUPS"

Two modifications were made:
 - add (define\s+)? which will match "define " but also an empty value.
   Thanks to this, the second group will always contain the variable
   name.
 - remove \s*(\+|)= which seems superfluous

Also, add GCC_TARGET in ALLOWED variable to avoid the following
warnings:

arch/arch.mk:12: possible typo: GCC_TARGET_ARCH -> *ARCH*
arch/arch.mk:13: possible typo: GCC_TARGET_ABI -> *ARCH*
arch/arch.mk:14: possible typo: GCC_TARGET_NAN -> *ARCH*
arch/arch.mk:15: possible typo: GCC_TARGET_FP32_MODE -> *ARCH*
arch/arch.mk:16: possible typo: GCC_TARGET_CPU -> *ARCH*
arch/arch.mk:17: possible typo: GCC_TARGET_FPU -> *ARCH*
arch/arch.mk:18: possible typo: GCC_TARGET_FLOAT_ABI -> *ARCH*
arch/arch.mk:19: possible typo: GCC_TARGET_MODE -> *ARCH*

Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-06 22:12:34 +01:00
James Hilliard
c3029878c5 utils/scanpypi: fix buildutils check in python3
The message attribute does not exist in python3, see PEP-0352:
https://www.python.org/dev/peps/pep-0352/

Fixes:
Traceback (most recent call last):
  File "utils/scanpypi", line 743, in <module>
    main()
  File "utils/scanpypi", line 693, in main
    if 'buildutils' in err.message:
AttributeError: 'ImportError' object has no attribute 'message'

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-06 22:12:11 +01:00
Thomas Petazzoni
0e64537917 utils/genrandconfig: reduce the maximum "size" of random configurations
genrandconfig is used by the Buildroot autobuilders to generate
semi-random configurations that we build test. As part of this, we use
"make randpackageconfig" to randomize the selection of packages,
together with a KCONFIG_PROBABILITY value, which indicates the
probabibility for each option to be enabled. This probability is
itself randomized, between 1% and 30% for every build.

However, with our increasing number of packages (over 2900), when we
use a 30% probability for options to be enabled, it means a *lot* of
options are enabled, causing very large configurations to be
tested. These configurations are not very realistic, and they take
ages to build on our autobuilders: we have builds that take 4, 5 or
even 7 hours to build.

In order to test a larger number of configurations and therefore a
larger variety of configurations, this commit reduces the maximum
probability to 20%.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-11-14 13:50:50 +01:00
Thomas Petazzoni
79cba4056b utils/getdeveloperlib.py: call Developer.hasfile() with relative path
In commit
40bb37bd70 ("utils/getdeveloperlib.py:
use relative paths for files"), the Developer class was changed to use
relative paths, including for its .hasfile() method.

However the check_developers() function of getdeveloperlib.py was not
updated accordingly, and continued to pass absolute paths. This caused
"get-developers -c" to return the entire list of files in Buildroot as
being unmaintained, as none of them were matching the file listed in
the DEVELOPERS file.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-02 23:20:11 +01:00
Thomas Petazzoni
53da6a7c05 utils/getdeveloperlib.py: fix check_output() return value decoding
In Python 3.x, check_output() returns a "bytes" array, and not a
string. Its result needs to be decoded to be turned into a
string. Without this fix, "get-developers -c" bails out with:

Traceback (most recent call last):
  File "/home/thomas/projets/buildroot/./utils/get-developers", line 105, in <module>
    __main__()
  File "/home/thomas/projets/buildroot/./utils/get-developers", line 53, in __main__
    files = getdeveloperlib.check_developers(devs)
  File "/home/thomas/projets/buildroot/utils/getdeveloperlib.py", line 280, in check_developers
    files = subprocess.check_output(cmd).strip().split("\n")
TypeError: a bytes-like object is required, not 'str'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-02 23:20:06 +01:00
Bernd Kuhls
8d469858ce utils/genrandconfig: test configurations with BR2_PACKAGE_PYTHON3_PY_ONLY
Python3 variant of
https://git.buildroot.net/buildroot-test/commit/?id=c0de21d9530af53eae5588d99d90c7e0cb87c543

to support Kodi 19 which depends on python3:
https://git.buildroot.net/buildroot/commit/?id=148e695e37561fe45d4726cb68f6454464d17797

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-10-18 22:10:08 +02:00
Petr Vorel
d50290764e utils/scanpypi: remove python2 compatibility imports
ee8b680816 ("utils/scanpypi: use python3 explicitly") started to use python3,
thus compatibility can be removed:

from __future__ import print_function
from __future__ import absolute_import

Tested with python3 -m py_compile.

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-22 21:40:43 +02:00
Petr Vorel
35f15db30a support: utils: use python3 explicitly
Python 2 is EOL sice 2020 [1], it's still available on distros, but may not
be installed by default (as being replaced by python3).

Thus remove compatibility imports:
from __future__ import print_function
from __future__ import absolute_import

Tested with python3 -m py_compile.

[1] https://www.python.org/doc/sunset-python-2/

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-22 21:30:24 +02:00
James Knight
eb75d71b80 utils/getdeveloperlib.py: force foward-slash for pattern matching
Force relative file path resolution of DEVELOPERS file entries to use
forward-slash separators since pattern matching assumes forward slashes.
This is to help permit uses invoking `get-developers` on Platforms where
`os.sep` may not be a forward slash.

Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-11 22:33:42 +02:00
James Knight
9f127cc420 utils/getdeveloperlib.py: explicitly set devs document encoding
Explicitly indicate the file encoding to UTF-8 for the DEVELOPERS
document. This prevents Unicode decoding errors when printing E-Mail
entries with Unicode characters on systems using an alternative default
encoding (e.g. 'CP1252').

This corrects the following observed error:

    $ ./utils/get-developers outgoing/*
    Traceback (most recent call last):
      File "utils\get-developers", line 105, in <module>
        __main__()
      File "utils\get-developers", line 47, in __main__
        devs = getdeveloperlib.parse_developers()
      File "...\buildroot\utils\getdeveloperlib.py", line 239, in parse_developers
        for line in f:
      File "...\Python<ver>\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6659: character maps to <undefined>

Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-09-05 16:08:10 +02:00
Yann E. MORIN
a30204b005 utils/test-pkg: add mode to only prepare .config files
Currently, running test-pkg is only done locally on the developers
machine.

In a follow up commit, we'll add the possibility to run test-pkg in a
gitlab-ci pipeline and, to speed things up, with one job per buildable
configuration.

As such, we will need to make sure that test-pkg only prepares the
configurations, and that it does not build them.

Add such a mode, with a new option, --prepare-only

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-28 17:01:11 +02:00
Yann E. MORIN
3418a068be utils/test-pkg: remove configurations that are skipped
When the config fragment provided by the user is not usable with a
specific toolchain configuration, the resulting .config file was kept
around.

In a follow up commit, we'll need to know, from outside test-pkg, if a
specific configuration was indeed usable or not.

So, unless if the user actually requested to keep the build directories,
remove the .config file when it contains a configration that would be
skipped.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-05 22:35:24 +02:00
Arnout Vandecappelle (Essensium/Mind)
f982f70434 utils/scanpypi: fix flake8 errors
Commit e43c050944 introduced two flake8
errors:

utils/scanpypi:300:26: E231 missing whitespace after ','
utils/scanpypi:302:9: F841 local variable 'setup' is assigned to but never used

The first one is easily fixed. The second one needs a little bit of
explanation. Before commit e43c0509, the return value of
imp.load_module() was used to be able to explicitly call the 'setup'
function in it in case the metadata was not populated. Since that
commit, calling that function is no longer needed, since setup.py is
executed in exactly the same way as when it's run from the command line,
so if that doesn't work, it's completely broken anyway. Therefore, we
can simply discard the return value of imp.load_module().

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-07-27 23:29:07 +02:00
Thomas De Schampheleire
e43c050944 utils/scanpypi: allow installation of commands without 'main' method
In case the setup.py file of a python package does not directly call the
'setup' method, utils/scanpypi was hoping there be a 'main' function which
would do the work, normally called via a construct like:

    if __name__ == '__main__':
        main()

However, this construct is nonstandard, and there are packages in PyPI which
call 'setup()' directly from the 'if' statement, without a main() method.

But scanpypi does not actually need to make such assumption: when loading
the module, it can decide the name to be '__main__', just as if setup.py
would be loaded interactively.

Additionally, remove some logic seemingly related to the previous trick of
calling 'main'. There should not be a problem in keeping already loaded
modules in sys.modules, as this is the purpose of sys.modules.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-07-25 23:43:13 +02:00
Thomas De Schampheleire
ad042904f4 utils/scanpypi: add setup.py script directory as sys.path[0]
Even though the directory containing a package's setup.py was added to
sys.path, some setup.py implementations rely on the fact that it is placed
in sys.path[0].

An example package is 'cram' which failed to be added with scanpypi:

    Traceback (most recent call last):
      File "utils/scanpypi", line 756, in <module>
        main()
      File "utils/scanpypi", line 703, in main
        package.load_setup()
      File "utils/scanpypi", line 303, in load_setup
        setup = imp.load_module('setup', s_file, s_path, s_desc)
      File "/usr/lib/python3.8/imp.py", line 234, in load_module
        return load_source(name, filename, file)
      File "/usr/lib/python3.8/imp.py", line 171, in load_source
        module = _load(spec)
      File "<frozen importlib._bootstrap>", line 702, in _load
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 783, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 44, in <module>
        long_description=long_description(),
      File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 20, in long_description
        return open(os.path.join(sys.path[0], 'README.rst')).read()
    FileNotFoundError: [Errno 2] No such file or directory: '.../buildroot/utils/README.rst'

The corresponding code from cram's setup.py is:

    def long_description():
        """Get the long description from the README"""
        return open(os.path.join(sys.path[0], 'README.rst')).read()

Indeed, the Python documentation says:

https://docs.python.org/3.8/library/sys.html#sys.path
    "...
    As initialized upon program startup, the first item of this list,
    path[0], is the directory containing the script that was used to invoke
    the Python interpreter.
    ..."

Fix this by inserting explicitly at index 0 instead of appending to
sys.path.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-07-25 23:43:11 +02:00
Fabrice Fontaine
60aa896904 utils/scanpypi: search LICENSE.MD
Some packages such as python-idna has a LICENSE.md file:
https://github.com/kjd/idna/blob/master/LICENSE.md

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-07-04 22:04:28 +02:00
Peter Korsgaard
8d07baab43 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-06-07 17:14:37 +02:00
Thomas De Schampheleire
7c9dfe4fd9 utils/genrandconfig: also test BR2_ENABLE_RUNTIME_DEBUG
... similar to BR2_ENABLE_DEBUG.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-06-01 22:40:12 +02:00
Matthew Weber
4e55bc8a4d utils/genrandconfig: drop hardening Config enables
Since 810ba387be, some form of these options are enable
by default. Specifically:

- Kept FORTIFY level 2 option as the default is now level 1.
- Removed all SSP options as the default now uses the best
  option based on toolchain support.
- Similar to SSP, for RELRO, the default now uses the best
  option based on toolchain support.
- Completely drop PIC PIE as it defaults =y

Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-24 15:29:26 +02:00
Thomas Petazzoni
ee8b680816 utils/scanpypi: use python3 explicitly
scanpypi is python3 compatible. In addition, it executes the setup.py
of Python modules to extract the relevant information. Since these are
more and more commonly using python3 constructs, using "python" to run
scanpypi causes problems on systems that have python2 installed as
python, when trying to parse setup.py scripts with python3 constructs.

Fixes part of #13516.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-20 17:42:46 +01:00
Thomas Petazzoni
38b0560f4e utils/getdeveloperlib.py: reduce Cc: list based on package infras
When a developer has package/pkg-<infra>.mk assigned to him/her in the
DEVELOPERS file, this has 3 implications:

 (1) Patches adding new packages using this infrastructure are Cc'ed
     to this developer. This is done by the analyze_patch() function,
     which matches the regexp r"^\+\$\(eval
     \$\((host-)?([^-]*)-package\)\)$" in the patch, i.e where an
     added line contains a reference to the infra maintained by the
     developer.

 (2) Patches touching the package/pkg-<infra>.mk file itself are Cc'ed
     to this developer.

 (3) Any patch touching a package using this infra are also Cc'ed to
     this developer.

Point (3) causes a significant amount of patches to be sent to
developers who have package/pkg-generic.mk and
package/pkg-autotools.mk assigned to them in the DEVELOPERS
file. Basically, all patches touching generic or autotools packages
get CC'ed to such developers, which causes a massive amount of patches
to be received.

So this patch adjusts the getdeveloperlib.py to drop point (3), but
preserves point (1) and (2). Indeed, it makes sense to be Cc'ed on new
package additions (to make a review that they use the package
infrastructure correctly), and it makes sense to be Cc'ed on patches
that touch the infrastructure code itself.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-10 22:10:58 +01:00
Francois Perrad
9e57d626c3 package/perl: bump to version 5.32.1
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-25 22:29:32 +01:00
Yann E. MORIN
9277978e28 utils: fix flake8 warning
Commit 40bb37bd70 refactored get-developers, and now the 'os' module is
no longer needed, but still imported:

    utils/get-developers:6:1: F401 'os' imported but unused
    1     F401 'os' imported but unused

Drop it now.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-01-23 21:24:48 +01:00
Thomas Petazzoni
40bb37bd70 utils/getdeveloperlib.py: use relative paths for files
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.

parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.

This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.

Before this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]

$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": []
}

After this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
  "Floris Bos <bos@je-eigen-domein.nl>",
  "Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": [
    "Christophe Priouzeau <christophe.priouzeau@st.com>"
  ]
}

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:08 +01:00
Thomas Petazzoni
57ecb6c8eb utils/get-developers: use Developers.hasfile() where appropriate
Instead of open-coding Developers.hasfile() in utils/get-developers,
use it directly.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:07 +01:00
Thomas De Schampheleire
a1bb132a81 utils/checkpackagelib/lib_mk.py: handle 'else' and 'elif' statements
An 'else' or 'elif' clause inside a make conditional should not be indented
in the same way as the if/endif clause. check-package did not recognize the
else statement and expected an indentation.

For example:

ifdef FOOBAR
	interesting
else
	more interesting
endif

would, according to check-package, need to become:

ifdef FOOBAR
	interesting
	else
	more interesting
endif

Treat 'else' and 'elif' the same as if-like keywords in the Indent test, but
take into account that 'else' is also valid shell, so we need to correctly
handle line continuation to prevent complaining about the 'else' in:

ifdef FOOBAR
	if true; \
	    ... \
	else \
	    ... \
	fi
endif

We don't add the 'else' and 'elif' statements to start_conditional, because
it would cause incorrect nesting counting in class OverriddenVariable.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-02 13:54:59 +01:00
Francois Perrad
b689a5a720 utils/scancpan: add GPL* as license file
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-29 23:00:53 +01:00
Francois Perrad
1120980f30 utils/scancpan: handles README.pod as license file
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-29 23:00:45 +01:00
Heiko Thiery
afc112b0e4 utils/getdeveloperlib.py: fix issue with hasfile()
pkg-stats is not able anymore to set the developers for defconfigs and
packages. This issue is introduced with
ae86067a15. The hasfile() method from
Developer object tries to check an absolute path against a relative path.

Convert the filepath to be checked also into an absolute path.

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-17 23:47:01 +01:00
Matt Weber
3930fd2ddd genrandconfig: uboot-tools env/scr creation test files
Normally the kconfig stings would end up empty and cause a build
error.  This patch provides test files to allow testing the creation
of uboot environment and script bin files from user provided txt files.

Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2020-10-06 23:37:48 +02:00
Thomas Petazzoni
f9150a6a3d utils/scanpypi: use raw strings in re.compile/re.sub
Fixes the following Python 3.x flake8 warning:

W605 invalid escape sequence '\w'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:56:17 +02:00
Thomas Petazzoni
c5b848d719 utils/getdeveloperlib.py: use raw strings for re.compile/re.match
Fixes the following Python 3.x flake8 warning:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:56:11 +02:00
Thomas Petazzoni
163f160a8e utils/{check-package, checkpackagelib}: consistently use raw strings for re.compile
Raw strings need to be used when calling re.compile() otherwise Python
3.x flake8 complains with:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-14 21:55:59 +02:00
Fabrice Fontaine
e15c35f588 utils/scancpan: use two spaces indentation in hash file
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-15 23:17:46 +01:00
Peter Korsgaard
fd99eb5016 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-09 15:17:09 +01:00
Peter Korsgaard
06417e97e3 utils/genrandconfig: drop outdated python-nfc check
Commit 9ea528f84b (package/python-nfc: bump to version 0.13.5) changed the
python-nfc package to download from github, so the package no longer needs
bzr on the host.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-04 19:08:12 +01:00
Peter Korsgaard
22e833af5e Config.in: drop BR2_NEEDS_HOST_{JAVAC,JAR}
With classpath removed, no packages select these symbols any more - So drop
them and their corresponding logic in dependencies.sh / genrandconfig.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-03 23:55:48 +01:00
James Hilliard
9fbbf4fd2a utils/scanpypi: update hash file indentation formatting
The new .hash convention is to use 2 spaces between fields.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-26 20:53:33 +01:00
Yann E. MORIN
76772cefd8 utils/check-package: ignore ACLOCAL_PATH
The aclocal program is provided by the automake package, so it makes
sense to define aclocal-related variables in automake.mk.

Add an exception to check-package to ignore that variable.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-04 17:15:19 +01:00
Francois Perrad
5bff3945af utils/scancpan: warn when a module is a perl core module
we don't want create new BR package with perl core module,
because core modules are already included in perl distribution,
and built with the BR package perl.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-08 18:16:54 +01:00