Commit Graph

265 Commits

Author SHA1 Message Date
Asaf Kahlon
143f135833 scanpypi: write every license file once
On some cases, when the package contains multiple license files
and some of them from the same type, the scanpypi script will write
the same license type more than once under _LICENSE.
Hence, before creating the _LICENSE variable, we'll remove every
possible duplication.

Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-11 23:04:40 +02:00
Heiko Thiery
5093435f66 utils/test-pkg: ensure to exit with an error upon failure
This commit modifies the main() function so that it returns the sum of
build and legal errors, making sure the overall test-pkg script exists
with a non-zero error code upon failure.

Signed-off-by: Heiko Thiery <heiko.thiery@kontron.com>
[Thomas: improved commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-10 22:54:12 +02:00
Jerzy Grzegorek
31fc6a1fe4 utils/checkpackagelib: CommentsMenusPackagesOrder: add more Config.in files to check
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: remove boot/Config.in, it is not ordered correctly yet.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
16271ec356 utils/checkpackagelib: CommentsMenusPackagesOrder: initialize 'menu_of_packages' array
'source' without a previous 'menu' is common in package/Config.in in
br2-externals.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
b47590fdb0 utils/checkpackagelib: CommentsMenusPackagesOrder: initialize in before()
This makes sure the state from a previous run (previous file) can never
leak over into the next file.

Also order the initializations alphabetically.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
6a5f6ea7e2 utils/checkpackagelib: CommentsMenusPackagesOrder: use '-' to describe state
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
0625899e3b utils/checkpackagelib: CommentsMenusPackagesOrder: separate comment/if/menu cases
The handling of 'comment...', 'if ...' and 'menu ...' lines have almost
nothing in common, and subsequent patches will give them even less in
common. Therefore, completely separate their handling in top-level
conditions. The only code that gets duplicated in the different branches
is the 'self.initialize_level_elements(text)' call.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
dd251d68e5 utils/checkpackagelib: CommentsMenusPackagesOrder: remove '-comment' state before the '-menu' one
A comment is considered an alternative delimiter like a menu. I.e.,
a menu that comes after a comment should not be considered a submenu of
that comment. Therefore, remove the '-comment' state before adding the
'-menu' one.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-07 23:39:21 +02:00
Jerzy Grzegorek
c82e0092f5 utils/checkpackagelib: CommentsMenusPackagesOrder: change the type of variable 'new_package'
Change the type of variable 'new_package' to make it a class member.
It will be used not only locally. Also initialize it.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
7197d3cb80 utils/checkpackagelib: CommentsMenusPackagesOrder: add functions to initialize arrays elements
Factor out two functions to initialize arrays elements. They will be
reused by followup patches.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
4e88b37274 utils/checkpackagelib: CommentsMenusPackagesOrder: get value of variable 'level'
Get value of variable 'level' only just after the state change.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
6748e42ace utils/checkpackagelib: CommentsMenusPackagesOrder: change the type of variable 'level'
Change the type of variable "level" to make it a class member.
It will be used not only locally.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: initialize self.level in the before() method, as suggested by
Ricardo]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:42 +02:00
Jerzy Grzegorek
dd99dc5763 utils/checkpackagelib: CommentsMenusPackagesOrder: rename variable 'm'
Rename variable 'm' for better readability.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:24:39 +02:00
Francois Perrad
f71f944fe6 utils/scancpan: improve license file detection
MANIFEST may contains line like this:
"LICENSE                     LICENSE file (added by Distar)"
so, retains only the first word.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-17 22:51:01 +02:00
Thomas Petazzoni
7d14a659db utils/genrandconfig: only do reproducible builds with diffoscope
After a few weeks of running reproducible builds in the autobuilders,
we found out that such builds without a diffoscope analysis are pretty
useless: the cmp on the tarballs doesn't help us fix the
reproducibility issue.

So, let's only do reproducible builds when diffoscope is available.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-07 16:34:44 +02:00
Thomas Petazzoni
4693676ec5 utils/genrandconfig: instantiate SystemInfo() earlier
In a future patch, we are going to need a SystemInfo() instance in
gen_config(), so create the SystemInfo() instance there, and pass it
to fixup_config(), where we currently use it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Atharva Lele <itsatharva@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-07 16:34:40 +02:00
Victor Huesca
a0a066a8ff utils/getdeveloperlib.py: fix flake8 warnings
Fix a weird blank-line missing and 2 trailing blank-space.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 22:22:01 +02:00
Victor Huesca
caead54274 utils/getdeveloperlib.py: add defconfig parsing
This patch extends the Developer class so that it associates each
developer with the defconfigs (in configs/) is in responsible for,
according to the DEVELOPERS file.

It will allow using the getdeveloperlib module to find which developer
is responsible for which defconfig, and send e-mail notifications of
defconfig build failures.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 00:02:35 +02:00
Victor Huesca
e635836494 utils/getdeveloperlib.py: add runtime test parsing
This patch extends the Developer class so that it associates each
developer with the runtime tests (in support/testing) is in
responsible for, according to the DEVELOPERS file.

The implementation relies on the unittest module to list all test
cases and does some manual parsing of these test-case objects to get
the actual list of test-cases per test-suite.

A global variable is used to compute the list of unittest only once.

This feature will allow to use the getdeveloperlib module to find
which developer is responsible for which runtime test, and send e-mail
notifications of runtime tests failures.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 00:02:09 +02:00
Victor Huesca
bb5576db9b utils/getdeveloperlib: add some debug information
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 23:39:55 +02:00
Ricardo Martincoski
75d6c2ba96 check-package: ignore external.mk
The external.mk file in a br2-external usually contains raw makefile
targets. This file is common code and not a package recipe so it should
not be tested against the code-style of a package .mk file.

When using this script to check files in a br2-external tree, usually
the user is responsible for not passing files that check-package do not
understand. But external.mk is special because it is part of the
br2-external structure, so it is likely someone expects it to be
checkable by an in-tree script.

Instead of adding another blob to the manual, just ignore this file.
Only do that when a br2-external is being tested (so with option -b
passed to the script) and also check that it is on the root path of the
br2-external to allow someone to have a package called external.

Reported on bug #11271.

Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski
1842bb1470 check-package: fix check of file in current dir with -b
One of the possible usages of check-package is to first cd to the
directory that contains the files to test (e.g. a package directory) and
then call the script passing the files in the current dir.
It already works when used for intree files, but for files in a
br2-external it throws an exception because some check functions (from
utils/checkpackagelib/lib_*.py) do need the name of the file being
processed and assume there will be a slash before the name.

Fix all check functions that assume that the full filename being checked
contains a slash. Do not use regexps to extract the filename, use
os.path functions instead.

Notice RemoveDefaultPackageSourceVariable and TypoInPackageVariable lead
to an exception in this case, but ApplyOrder instead generates a false
warning.

Fixes bug #11271.

Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Arnout Vandecappelle (Essensium/Mind)
d5990601ba utils/checkpackagelib: CommentsMenusPackagesOrder: properly initialize levels
Fix an issue introduced by Arnout while committing. Jerzy originally
initialized the menu_of_packages, package and print_package_warning
members like they should be, but Arnout thought it wasn't needed and
removed that.

It is actually needed, to make sure the top level (level 0) works.

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/264383157

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-02 18:43:52 +02:00
Arnout Vandecappelle (Essensium/Mind)
0a4e1fc054 utils/checkpackagelib: CommentsMenusPackagesOrder: use regex for source
The 'source' strings identify which package is incorrectly ordered. We
need to extract the actual package name from that string, which is
currently done with constants that assume the file is package/Config.in.

In addition, only 'source' lines that are indented with a tab are
checked. This kind of indentation is done in package/Config.in, but not
e.g. boot/Config.in.

Therefore, use a regular expression to match the 'source' lines, and to
extract the directory part from it.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Jerzy Grzegorek
83a34f7705 utils/checkpackagelib: CommentsMenusPackagesOrder: append elements to arrays if needed
In the future, the nesting level of menus, comments and conditions may
increase. The fixed array length used now is not appropriate. Therefore,
append elements to the arrays if needed.

Also change order of variables.

Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Arnout Vandecappelle (Essensium/Mind)
c62a282920 utils/checkpackagelib: CommentsMenusPackagesOrder: fix 'menuconfig' handling
The CommentsMenusPackagesOrder check builds the 'state' to track the
depth of menus and conditions. However, a menuconfig doesn't create a
menu by itself - it is always followed by a condition that implies the
menu. As a result, when unwinding the 'state', the level will be wrong.

Fix this by checking for menu followed by a space, so it no longer
matches menuconfig. For consistency, do the same for comment and if
as well.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 22:48:59 +02:00
Arnout Vandecappelle (Essensium/Mind)
5603406bd1 utils/checkpackagelib: CommentsMenusPackagesOrder: only apply to top-level
The CommentsMenusPackagesOrder test is broken in various ways for files
other than package/Config.in and package/Config.in.host. Therefore, the
script gives bogus errors for various other Config.in files.

However, we don't really want to check those other files. Indeed, many
of them have a non-alphabetical ordering for good reasons.

Therefore, skip the check for files other than package/Config.in and
package/Config.in.host.

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/251214899

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-08-01 09:58:15 +02:00
Jerzy Grzegorek
e36a63cf6b checkpackagelib/lib_config.py: check packages alphabetical order in {Config.in, Config.in.host}
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
[Arnout:
 - calculate level by counting - instead of with a static array;
 - new_package is only used locally, so don't make it a class member;
 - do indentation according to length of prefix;
 - don't split string in the middle of a line;
 - report first wrong package per menu;
 - do replace() only once;
 - add comment why we do replace().
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-07-14 00:21:26 +02:00
Brandon Maier
44af8386f8 utils/check-package: Add a quiet flag
When running in a CI system, stat messages become white noise. Introduce
an option to suppress non-error, non-warning, messages.

Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-07-03 23:34:58 +02:00
Thomas De Schampheleire
72bf48606c utils/test-pkg: clean output dir for successful builds
test-pkg will use gigabytes of space when testing all toolchains.
Nevertheless, you are normally only interested in the actual build / host
tree when there is a build failure.

Do a 'make clean' for successful builds to save disk space, unless the new
option '-k/--keep' is set.
Note that the logfile and configuration is always retained for inspection.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-23 21:39:08 +02:00
Thomas De Schampheleire
989cda12ba utils/test-pkg: fix long option parsing
The long option parsing of test-pkg is broken because:
- some long options are not declared
- there should be a comma between long options, the colon does not replace
it.

This change also revealed that the declaration of 'toolchains-dir' should
have been 'toolchains-csv', originally introduced in commit ed59f81a3c.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-23 21:38:13 +02:00
Francois Perrad
280149ce60 utils/scancpan: fix test file creation on br-external
first, add the creation of the needed subdirectories

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-09 23:37:04 +02:00
Atharva Lele
9355933afd utils/genrandconfing: randomly enable BR2_REPRODUCIBLE 10% of the times
Initial patch to start reproducibility testing.

Also enable tar filesystem to compare images since diffoscope works well
with tar.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-08 22:56:41 +02:00
Peter Seiderer
b9b081d53a utils/check-package: fix flake8 warning
Fix flake 8 warning (introduced with commit [1]):

  utils/checkpackagelib/lib.py:56:1: E302 expected 2 blank lines, found 1

[1] https://git.buildroot.net/buildroot/commit/?id=8e352c32b0beded97a8a5c1e9edc9d618514ee7b

Reported-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-20 10:20:38 +02:00
Peter Seiderer
8e352c32b0 utils/check-package: warn about utf-8 characters in .mk files
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-18 23:34:59 +02:00
Thomas Petazzoni
2e1f98d950 utils/genrandconfig: prevent selecting libffi on nds32
libffi does not have support for the nds32 architecture, so let's
disallow libffi on this CPU architecture for the moment.

Fixes:

  http://autobuild.buildroot.net/results/12803a705586e82fdfb49013da2eb3b9879ccd45/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-17 21:17:42 +02:00
Thomas Petazzoni
472e8808d9 utils/genrandconfig: test configurations with BR2_SYSTEM_ENABLE_NLS
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-14 11:26:58 +02:00
Peter Korsgaard
83f82bd67a utils/getdeveloperlib.py: print warnings/errors to stderr
Instead of stdout where it gets mixed with the normal output, confusing
software parsing the output (E.G. get-developers -e as git sendemail.ccCmd).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-04 22:54:53 +02:00
Francois Perrad
f4a30ae5bd utils/scancpan: add a check of XS dependencies in generated test
The XS modules have a high propability to fail (compared to Pure Perl modules),
so it is valuable to check XS dependencies before the check of the main module.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-31 14:01:36 +02:00
Vadim Kochan
a946813dd5 utils/test-pkg: generate package config if it is not specified
It is possible to generate one-line config for the package just by
normalize it to the form:

    BR2_PACKAGE_${pkg_replaced-to_and_uppercase}

it simplifes a bit of testing package where no additional config options
are needed.

Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 10:03:34 +01:00
James Hilliard
f13b843e71 utils/scanpypi: handle underscores in python packages
Some python packages seem to use underscores in inconsistent ways.  We can
attempt to normalize these by always using dashes for the buildroot name and
attempting to autodetect the correct metadata name format.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 12:37:48 +01:00
Peter Korsgaard
a83e30ad63 utils/scanpypi: protect against zip-slip vulnerability in zip/tar handling
For details, see https://github.com/snyk/zip-slip-vulnerability

Older python versions do not validate that the extracted files are inside
the target directory.  Detect and error out on evil paths before extracting
.zip / .tar file.

Given the scope of this (zip issue was fixed in python 2.7.4, released
2013-04-06, scanpypi is only used by a developer when adding a new python
package), the security impact is fairly minimal, but it is good to get it
fixed anyway.

Reported-by: Bas van Schaik <security-reports@semmle.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-12 21:27:35 +01:00
Peter Korsgaard
a11d61e104 utils/genrandconfig: drop qt4 fixups
Now that Qt4 is removed, these fixups can be removed as well.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 16:30:39 +01:00
Ricardo Martincoski
b03fa5d96f utils/check-package: warn about overridden variables
For the general case, appending values to variables is OK and also a
good practice, like this:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR += value2

or this, when the above is not possible:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR := $(PACKAGE_VAR), value2

But this override is an error:
|PACKAGE_VAR = value1
|PACKAGE_VAR = value2

as well this one:
|ifeq ...
|PACKAGE_VAR += value1
|endif
|PACKAGE_VAR = value2

And this override is error-prone:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR = value2

Create a check function to warn about overridden variables.

Some variables are likely to have a default value that gets overridden
in a conditional, so ignore them. The name of such variables end in
_ARCH, _CPU, _SITE, _SOURCE or _VERSION.

After ignoring these variable names, there are a few exceptions to this
rule in the tree. For them use the comment that disables the check.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Simon Dawson <spdawson@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 20:24:57 +01:00
Ricardo Martincoski
fe7a5d7120 utils/check-package: handle ifdef/ifndef in .mk files
Currently check-package only knows about ifeq/ifneq.
Add code to handle ifdef/ifndef as well.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-29 16:38:41 +01:00
Ricardo Martincoski
4a6c5ab2c3 utils/check-package: allow to disable warning for a line
Currently any exceptions for a check function need to be coded into the
check-package script itself.

Create a pattern that can be used in a comment to make check-package
ignore one or more warning types in the line immediately below:
 # check-package Indent, VariableWithBraces

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-29 16:37:47 +01:00
Ricardo Martincoski
02b165dc71 check-package: fix Python3 support
This script currently uses "/usr/bin/env python" as shebang but it does
not really support Python3. Instead of limiting the script to Python2,
fix it to support both versions.

So change all imports to absolute imports because Python3 follows PEP328
and dropped implicit relative imports.

In order to avoid errors when decoding files with the default 'utf-8'
codec, use errors="surrogateescape" when opening files, the docs for
open() states: "This is useful for processing files in an unknown
encoding.". This argument is not compatible with Python2 open() so
import 'six' to use it only when running in Python3.
As a consequence the file handler becomes explicit, so use it to close()
the file after it got processed.

This "surrogateescape" is a simple alternative to the complete solution
of opening files with "rb" and changing all functions in the lib*.py
files to use bytes objects instead of strings. The only case we can have
non-ascii/non-utf-8 files being checked by the script are for patch
files when the upstream file to be patched is not ascii or utf-8. There
is currently one case in the tree:
package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-16 23:14:25 +01:00
Peter Korsgaard
45aabcddc5 utils/get-developers: really make it callable from elsewhere than the toplevel directory
Commit 62d5558f76 (utils/get-developers: make it callable from elsewhere
than the toplevel directory) tried to fix this by passing in the toplevel
directory when the DEVELOPERS file is parsed.

Unfortunately this is not enough, as E.G.  also the paths listed in the
patches are relative to the toplevel directory, causing it to not match the
entries in the DEVELOPERS file.

In concept this can be fixed by also passing the toplevel directory to the
Developers class, but the simplest solution is just to chdir to the toplevel
Buildroot directory before calling any of the getdeveloperlib functions.

This does require us to finish parsing command line arguments (which opens
the provided patch files) to not get into trouble with relative paths to
patches before chdir'ing / initializing getdeveloperlib.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-04 13:23:41 +01:00
Peter Korsgaard
74fbc20504 Revert "utils/get-developers: make it callable from elsewhere than the toplevel directory"
This reverts commit 62d5558f76.

This actually does not work, as patches contain paths relative to the
toplevel directory as well.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-04 13:23:33 +01:00
Adam Duskett
bbf32a77ec utils/test-pkg: force checking dependencies
Currently, if a user runs "make" while specifying a specific package
(IE: make -p foo),  the Makefile logic skips checking to see if all the
dependencies are selected in the specified packages config file. This behavior
is useful to test simple packages which do not have "complex" dependencies.

However; if a developer uses test-pkg -p ${package_name} to check their package,
the package may pass all the checks, but would have otherwise failed with a
simple "make" because the developer may have failed to add a select line in
packages config file, even if there is a new dependency in the packages
Makefile.

Pass the environment variable "BR_FORCE_CHECK_DEPENDENCIES"  to the Makefile in
the test-pkg script,  and check it's value in the Makefile. If the value is
"YES" force checking for dependency issues.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-03 11:41:26 +01:00
Asaf Kahlon
2bcc4edceb utils/scanpypi: write _SOURCE only when needed
For some packages, there's no need to add the _SOURCE variable, since
the name of the source file is the same as the name of the package
(like python-engineio). Hence, we'll add it to the .mk file only if
needed.

Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-30 17:42:32 +01:00
Yegor Yefremov
e273c36ad0 utils/scanpypi: import third party modules after the standard ones
Move imports from six package after the standard modules. Resolves
pylint warnings.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-10 11:46:36 +01:00
Francois Perrad
84179d4a04 utils/scancpan: allow the generation of more tests
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:20:10 +01:00
Francois Perrad
1b9f4addbc utils/scancpan: mark XS modules in docstring of test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:19:48 +01:00
Thomas Petazzoni
e6a10b0284 utils/genrandconfig: fix flake8 warnings
Fixes:

utils/genrandconfig:369:17: E231 missing whitespace after ','
utils/genrandconfig:370:1: E101 indentation contains mixed spaces and tabs
utils/genrandconfig:370:1: W191 indentation contains tabs
utils/genrandconfig:372:1: E101 indentation contains mixed spaces and tabs

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 22:25:42 +01:00
Evgeniy Didin
aa556e2035 utils/genrandconfig: test with BR2_OPTIMIZE_2=y
Currently all random defconfigs which are used in autobuilder use size
optimizaion (-Os), since BR2_OPTIMIZE_S=y is the default.

Adding "-O2" optimization will give better test coverage.

In many cases software gets built with speed optimization rather than
size optimization. So let's add Level 2 optimizaion option to be
generated in random defconfigs, so we could be able to test how
packages are built with "-O2" in autobuilder.

Signed-off-by: Evgeniy Didin <Evgeniy.Didin@synopsys.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: arc-buildroot@synopsys.com
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 22:06:12 +01:00
Francois Perrad
65a42f26c8 utils/scancpan: add generation of test
This commit extends the scancpan script to automatically generate a
test for the Perl module, either if the Perl module uses native
library, or if it has more than one dependency.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:49:24 +01:00
Francois Perrad
6f94a68bb2 package/perl: bump to version 5.28.1
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 13:30:29 +01:00
Peter Korsgaard
13c43455a0 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-02 08:16:10 +01:00
Nasser Afshin
8ca206e5a1 utils/test-pkg: use the correct config prefix when merging
We should use an empty prefix as we do not have any prefix.
Note that BR2_ is mere a convention.

Signed-off-by: Nasser Afshin <Afshin.Nasser@gmail.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 10:11:13 +01:00
Thomas Petazzoni
3eb49f59d6 utils/genrandconfig: add missing new line when creating the configuration
When adding the custom BR2_WGET value in the configuration,
genrandconfig forgets to add a newline. Due to this, the next option
that is added is printed on the same line as BR2_WGET="", which causes
it to be ignored.

Due to this, in all builds, the line right after BR2_WGET was
ignored. It could have been BR2_ENABLE_DEBUG, BR2_INIT_BUSYBOX,
BR2_INIT_SYSTEMD, BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV,
BR2_STATIC_LIBS or BR2_PACKAGE_PYTHON_PY_ONLY depending on the
randomization.

Fix that by adding a proper newline at the end of the BR2_WGET option.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-18 08:37:11 +01:00
Yegor Yefremov
fd29797f65 utils/scanpypi: use archive file name to specify the extraction folder
Some packages have archive name that is different from package name.
For example websocket-client's archive name is websocket_client-*.tar.gz.
scanpypi expects the temporary extract folder to be:

/tmp-folder/BR-package-name/PyPI-packagename-and-version

In the case of websocket-client package the real extraction folder
will be different from the expected one because of the '_' in the
archive file name.

Use archive file name instead of package name to specify the extraction
folder. As the version is already part of this file, we don't need to
specify it.

Bonus: remove obsolete "return None, None" as the function doesn't return
anything. OSError class doesn't provide "message" member, so replace it
with "strerror".

Fixes:
https://bugs.busybox.net/show_bug.cgi?id=11251

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Asaf Kahlon <asafka7@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-02 21:35:08 +01:00
Francois Perrad
f95d44d3f3 utils/scancpan: print package/Config.in only when useful
Currently, utils/scancpan always outputs what should be placed in
package/Config.in to include all Perl packages Config.in
files. However, in practice, this is only useful when a new package is
added. This commit adjusts this behavior so that what should be place
in package/Config for Perl packages is only displayed when scancpan
has produced a new Buildroot package for a Perl module.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 22:21:30 +01:00
Francois Perrad
ccd05f6fe5 utils/scancpan: add a full stop to the help text when missing
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 14:41:00 +01:00
Francois Perrad
5b7563f3d6 scancpan: also use README.md as default license file
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-28 15:16:07 +01:00
Francois Perrad
fe12a66bde scancpan: add PERL_FOO_DISTNAME in perl-foo.mk
This variable is needed for upgrading packages with scancpan.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-10-24 00:22:13 +01:00
Marcel Patzlaff
bf9ccfc37b utils/diffconfig: remove BR2_* prefix restriction
The utils/diffconfig script works only on variables with the BR2_
prefix. This is OK for Buildroot [def]configs since this is the prefix
for all user-facing variables, but it prevents using the same script
to compare configs from kconfig-based packages.

Remove the BR2_ restriction, allowing usage such as:

  ./utils/diffconfig \
	board/qemu/xtensa-lx60/linux.config \
	board/qemu/xtensa-lx60/linux-nommu.config

Signed-off-by: Marcel Patzlaff <m.patzlaff@pilz.de>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 19:57:06 +02:00
Francois Perrad
0cae150f9e scancpan: remove optional dependency
there are also runtime dependency

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-20 19:29:52 +02:00
Peter Korsgaard
62d5558f76 utils/get-developers: make it callable from elsewhere than the toplevel directory
get-developers tries to open DEVELOPERS in the current directory, so it
breaks when calling it from elsewhere than the toplevel Buildroot directory.

Traceback (most recent call last):
  File "../utils/get-developers", line 107, in <module>
    __main__()
  File "../utils/get-developers", line 26, in __main__
    devs = getdeveloperlib.parse_developers(os.path.dirname()
  File "/home/peko/source/buildroot/utils/getdeveloperlib.py", line 161, in parse_developers
    with open(os.path.join(basepath, "DEVELOPERS"), "r") as f:
IOError: [Errno 2] No such file or directory: '/home/peko/source/buildroot/output-foo/DEVELOPERS'

Fix it by instead figuring out where the DEVELOPERS file is relative to the
location of get-developers (E.G. one level up).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[Arnout:
  - add realpath to support a symlinked get-developers script;
  - pass devs_dir argument to check_developers() to support -c in subdir;
  - convert basepath to absolute path to support -f option.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-10-20 17:24:13 +01:00
Grégoire Delattre
8320ad3341 utils/get-developers: fix python 3.x compatibility
This fixes a syntax error introduced in bcf2ed5cc3.

Output before the patch:
    $ ./utils/get-developers outgoing/*
        File "./utils/get-developers", line 97
            print dev
                    ^
        SyntaxError: Missing parentheses in call to 'print'. Did you mean
        print(dev)?

Output after the patch:
    $ ./utils/get-developers outgoing/*
    git send-email --to buildroot@buildroot.org

Signed-off-by: Grégoire Delattre <gregoire.delattre@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-12 09:05:50 +02:00
Yegor Yefremov
d05e41eb1a scanpypi: improve BSD licence handling
When used without spdx_lookup the BSD licence cannot be
detected correctly because many Python packages just specify
BSD without the exact version in their metadata. So add a
special message warning the user instead of the licence id.

Bonus: fix typo.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 09:32:33 +02:00
Francois Perrad
fd4d596f3d scancpan: generate a conventional comment
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 21:21:02 +02:00
Francois Perrad
4f7441ab41 scancpan: remove run-time dependencies
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:06:09 +02:00
Francois Perrad
37e85660b5 scancpan: split multiple dependencies on multiple lines
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:06:04 +02:00
Francois Perrad
b4fd962c46 scancpan: sort license in hash file
now, the output is reproductible

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:05:53 +02:00
Francois Perrad
8c3395a374 scancpan: sort license files
so, the output is reproductible

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:05:49 +02:00
Francois Perrad
8966be9d21 scancpan: trim the abstract
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:05:45 +02:00
Francois Perrad
228b6b680a scancpan: wrap abstract if too long
[note: code style fixed]

Signed-off-by: Christopher McCrory <chrismcc@gmail.com>
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 09:05:41 +02:00
Peter Korsgaard
bcf2ed5cc3 utils/get-developers: add -e flag to only list email addresses for git send-email
When called with a list of patches, get-developers prints the entire git
send-email invocation line:

./utils/get-developers 0001-git-security-bump-to-version-2.16.5.patch
git send-email --to buildroot@buildroot.org --cc "Matt Weber <matthew.weber@rockwellcollins.com>"

This may be handy when creating an entire patch series and editing a cover
letter, but it does mean that this has to be explicitly executed and
get-developers cannot be used directly by the --cc-cmd option of git
send-email to automatically CC affected developers.

So add an -e flag to only let get-developers print the email addresses of
the affected developers in the one-email-per-line format expected by git
send-email, similar to how get_maintainer.pl works in the Linux kernel.

With this and a suitable git configuration:

git config sendemail.to buildroot@buildroot.org
git config sendemail.ccCmd "$(pwd)/utils/get-developers -e"

You can simply do:

git send-email master

To automatically mail the buildroot list and CC affected developers on
patches.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-08 17:47:00 +02:00
Francois Perrad
6b9b8f1063 perl-file-sharedir-install: new package
Only the host variant is added, needed as a build time dependency of
the target perl-file-sharedir package.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-08 16:40:05 +02:00
Francois Perrad
c8136d8fb2 perl-extutils-installpaths: new package
Only the host variant is added, which is needed as a dependency of
perl-module-build-tiny (host), itself needed for
perl-apache-logformat-compiler (target).

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-08 12:09:43 +02:00
Francois Perrad
fe66b114a2 perl-extutils-helpers: new package
Only the host variant is added, which is needed as a dependency of
perl-module-build-tiny (host), itself needed for
perl-apache-logformat-compiler (target).

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-08 12:07:58 +02:00
Francois Perrad
efab462124 perl-extutils-config: new package
Only the host variant is added, which is needed as a dependency of
perl-module-build-tiny (host), itself needed for
perl-apache-logformat-compiler (target).

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-08 12:06:01 +02:00
Francois Perrad
547ba3a187 utils/scancpan: refactor license name adjustment
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-25 22:57:54 +02:00
Ricardo Martincoski
910ee6f7bd utils/check-package: detect the use of ${} in .mk files
And warn to use $() instead.
For examples see [1] and [2].

In the regexp, search for ${VARIABLE} but:
 - ignore comments;
 - ignore variables to be expanded by the shell "$${}".

[1] http://lists.busybox.net/pipermail/buildroot/2018-July/225211.html
[2] 36305380db

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-21 00:05:55 +02:00
Christopher McCrory
e9963cd2b5 scancpan: skip Dist::Zilla
Signed-off-by: Christopher McCrory <chrismcc@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-21 00:02:11 +02:00
Christopher McCrory
b8d768b247 utils/scancpan: support more licenses
Signed-off-by: Christopher McCrory <chrismcc@gmail.com>
Reviewed-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-21 00:01:13 +02:00
Yegor Yefremov
0101ac62c8 scanpypi: place a warning into *.mk file if licence id couldn't be detected
If a license file could be found, but license id couldn't be detected place
following warning into *.mk file:

FOO_LICENSE = FIXME: license id couldn't be detected

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-31 23:12:57 +02:00
Francois Perrad
cd87d05f77 scancpan: warn on undefined LICENSE
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-14 15:37:14 +02:00
Francois Perrad
a96680891a scancpan: warn on suspicious host dependencies
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-14 15:37:14 +02:00
Yann E. MORIN
2664022787 utils/test-pkg: log the output of merge-config
Currently, the stdout is consigned to oblivion, while the stderr is not
redirected at all. So, when the configuration under test redefines a
symbol, like is the case when testing busybox for example, there is an
ugly warning.

So, just redirect both stdout and stderr to the logfile.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-07-10 23:01:47 +02:00
Ofer Heifetz
78968330e3 support/config-fragments/autobuild: add Linaro AArch64 BE support
Signed-off-by: Ofer Heifetz <oferh@marvell.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-28 22:15:57 +02:00
Yegor Yefremov
fb775f4c87 scanpypi: rework download_package error handling
Some packages don't provide source archive but only a wheel file. In
this case download variable is not defined. So define this variable at
the very beginning and check whether it is None after searching for
source archives in the metadata.

Bonus: fix PEP8 issue with wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-06-15 14:06:43 +02:00
Francois Perrad
a12499bf99 utils/scancpan: generates hashes of license files
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-13 22:41:13 +02:00
Yegor Yefremov
6766ff9d12 scanpypi: add support for the new PyPI infrastructure
https://pypi.python.org URL has been changed to https://pypi.org.

Package's JSON object now contains sha256 checksum, so use it
instead of locally computed one. Change comments in the hash
file accordingly.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-18 17:11:47 +02:00
Thomas Petazzoni
d13c3d178c support/config-fragments/autobuild: remove Blackfin configuration
In preparation for the removal of the Blackfin architecture, drop the
autobuilder toolchain configuration that was testing Blackfin.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-15 22:03:18 +02:00
Arnout Vandecappelle (Essensium/Mind)
b3a5ab4c7a check-package: support symlinks to the script
The check-package script finds base_dir (= the Buildroot directory) and
cd's into it. To be able to support relative paths as arguments, it
first recalculates the arguments relative to base_dir.

However, if there is a symlink anywhere on the path to the
check-package script, the relative paths will be wrong. To solve this,
use realpath() instead of abspath(), so symlinks are resolved before
calculating the relative path.

Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 22:41:54 +02:00
Thomas Petazzoni
0534da0f9f test-pkg: test a subset of toolchains by default, add -a and -n options
During the latest Buildroot Developers meeting, we discussed that
test-pkg would perhaps be more widely used if it tested a smaller
subset of toolchains. Indeed, it currently tests 47 toolchains, which
takes very long to build. Several of the toolchain configurations are
quite similar, and it is perhaps not necessary for contributors to
test them all before submitting a package.

Therefore, this commit changes the test-pkg script to only test a
subset of the toolchain configurations by default. The N first
configurations of the CSV files are tested, where N is hard-coded in
the script. The CSV file has therefore been re-organized to have the
first N toolchains be the most important ones.

A -a/--all option is added to test with all toolchains, while a
-n/--number option is added to test with the first N toolchains, N
being passed on the command line.

Note that the list of toolchains (built in the "toolchains" shell
variable) is no longer sorted. Indeed, when the first N toolchains are
tested, we want them to be tested in the same order as they are listed
in the CSV file, as we are careful to order them in an interesting
order. We only sort when all toolchains are tested.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 16:30:34 +02:00
Thomas Petazzoni
73dcbc19e9 utils/checkpackagelib: exclude four files from Config.in indentation check
package/Config.in, package/Config.in.host, package/x11r7/Config.in and
package/kodi/Config.in do not comply with the normal Config.in
indentation rules. However, this violation of the rule is legitimate, so
let's skip them in check-package for this specific indentation check.

This removes the last 2197 remaining warnings on Config.in files.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Ricardo: rebase patch to use relative paths passed by the main script,
          fix flake8 warnings, add package/Config.* to the list]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 10:16:21 +02:00
Ricardo Martincoski
3ff0728480 scanpypi: generate help text compliant to check-package
Each line must fit in <tab><2 spaces><62 chars>.
The default width for textwrap.wrap() is 70, so explicit set it to 62.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 10:16:09 +02:00
Ricardo Martincoski
e2c4a687c7 check-package: enable for Config.in* in base dir
These files can benefit from this script to prevent common mistakes when
submitting patches.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 09:12:00 +02:00
Ricardo Martincoski
bfc8dce467 check-package: enable for toolchain/
The toolchain directory can benefit from this script to prevent common
mistakes when submitting patches.

In order to accomplish this:
Do not ignore anymore files from the toolchain/ directory.
Ignore this symbol:
 - BR_LIBC: defined by the buildroot toolchain, used by gcc-final.mk.

Ignore toolchain/toolchain-external/pkg-toolchain-external.mk as it
declares a package infra and not a package itself.
Ignore toolchain/helpers.mk as it contains only helper functions.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 09:09:01 +02:00
Ricardo Martincoski
4d39f5876d check-package: enable for linux/
This directory can benefit from this script to prevent common mistakes
when submitting patches.

In order to accomplish this:
Do not ignore anymore files from the linux/ directory.
Ignore missing LINUX_EXT_ prefix as the variables for linux extensions
do not use it.
Ignore this symbol:
 - LINUX_EXTENSIONS: defined by each linux extension, used by
   linux/linux.mk.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:54:07 +02:00
Ricardo Martincoski
0c83673638 check-package: enable for fs/
The filesystem types can benefit from this script to prevent common
mistakes when submitting patches.

In order to accomplish this:
Do not ignore anymore files from the fs/ directory.
Ignore fs/common.mk as it declares a package infra and not a package itself.
Register the ROOTFS_ as a valid prefix for variables.
Ignore these symbols:
 - PACKAGES_PERMISSIONS_TABLE: defined either by packages through
   pkg-generic or by filesystem types, used by fs/common.mk;
 - SUMTOOL: defined by package mtd, used by filesystem jffs2;
 - TARGETS_ROOTFS: defined by filesystem types, used in the main
   Makefile.

Keep using loose checks that warn about common mistakes while keep the
code simple.
As a consequence the check functions do not differentiate between
packages and filesystems so the symbol PACKAGE_UBI would not generate a
warning for the ubi filesystem neither the symbol ROOTFS_MTD would
generate a warning for the mtd package. But those kind of mistakes are
not common and are obvious in the code review, unlike typos i.e.
ROOTFS_UBl or PACKAGE_MID that would be hard to see in the code review.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:52:45 +02:00
Ricardo Martincoski
79936be11a check-package: enable for boot/
This directory can benefit from this script to prevent common mistakes
when submitting patches.

In order to accomplish this:
Do not ignore anymore files from the boot/ directory.
Ignore boot/barebox/barebox.mk as it declares a package infra and not a
package itself.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:49:18 +02:00
Ricardo Martincoski
560ab2cb95 check-package: enable for arch/ and system/
These directories can benefit from this script to prevent common
mistakes when submitting patches.

In order to accomplish this:
Do not ignore anymore files from these directories.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:01:05 +02:00
Ricardo Martincoski
a7b4bbfc14 check-package: prepare to extend to other directories
Currently the script only checks files inside the package/ directory.
Upcoming patches will enable it for other directories.

In order to reliably test for file names, i.e. the Config.in in the base
directory, normalize the path of files to check to a relative path to
the base directory.

Rename the variable that holds the compiled regexp to better represent
its content and rearrange how it is declared to make easy to later add
new directories to check. As a consequence the files that declare
package infra types would not be ignored anymore, so create a new
variable to list the files intree to be ignored during the check. The
same variable will be used by upcoming patches to ignore other files.
Ignore pkg-*.mk and doc-asciidoc.mk since they are package infra files.

In order to not produce weird results when used for files outside the
tree (i.e. in a private br2-external) add an explicit command line
option (-b) that bypasses any checks that would make a file be ignored
by the path that contains it.
When in this out-of-tree mode, the user is responsible for providing a
list of files to check that do not contain files the script does not
understand, e.g. package infra files.

As a result of this patch, besides the known use:
$ ./utils/check-package package/new-package/*
someone with the utils/ directory in the path can now also run:
$ cd package/new-package/
$ check-package *
or
$ check-package -b /path/to/br2-ext-tree/package/staging-package/*

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 07:58:07 +02:00
Peter Korsgaard
1490558b75 utils/scanpypi: don't hardcode python2
Commit 3a0c20c530 (scanpypi: add support for Python3) adapted the script
to work with python 3.x, but the shebang still said python2 making it
unlikely to work on systems without python 2.x.

Change it to just 'python' instead.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 07:53:58 +02:00
Nicolas Cavallari
42be6b6153 check-package: Flag usage of '\t \\$'.
check-package would flag tabs before a backslash ('\t\\'),
two spaces before a backslash ('  \\') but would not flag a tab before space
before backslash ('\t \\'), allowing someone to bypass the check.

Signed-off-by: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-03-31 23:02:14 +02:00
Thomas Petazzoni
0866a280e4 utils/genrandconfig: use --no-check-certificate in wget by default
A number of autobuilder failures are due to the fact that autobuilder
instances use old distributions, with old SSL certificates, and
therefore wget aborts with an error "The certificate of `xyz.org' is
not trusted.".

In order to avoid such failures that are not very interesting in the
context of the autobuilders, we pass --no-check-certificate to
wget. The integrity of the downloaded files is anyway verified by the
hashes, and this is only meant to be used in the context of
testing/CI, not in production.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-03-31 17:17:51 +02:00
Thomas Petazzoni
a86b2523a0 utils/genrandconfig: adjust to core-dependencies removal
In commit ea9669fffa ("core: kill
DEPENDENCIES_HOST_PREREQ"), the core-dependencies make target was
removed, and is now named just "dependencies".

This broke the utils/genrandconfig script, and this commit intends to
fix that.

Since this script is part of the Buildroot tree, it is provided
together with Buildroot, so we don't need to support the legacy
core-dependencies target. Someone checking out an older Buildroot will
have a working setup, with support/dependencies exposing
core-dependencies and utils/genrandconfig using core-dependencies. The
only broken situation will be between
ea9669fffa and this commit, but that's
not a sufficient reason to add some backward compatibility code.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-03-30 22:00:08 +02:00
Ricardo Martincoski
ede92be10d scanpypi: fix code style
Fix these warnings:
E401 multiple imports on one line

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-13 22:32:01 +01:00
Ricardo Martincoski
19aaf8c3f1 size-stats-compare: fix code style
Fix these warnings:
E129 visually indented line with same indent as next logical line
E302 expected 2 blank lines, found 1

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-13 22:27:19 +01:00
Ricardo Martincoski
f6809dba0f genrandconfig: fix code style
Fix these warnings:
E201 whitespace after '['
E202 whitespace before ']'
E501 line too long (138 > 132 characters)
 -> isolate the common part of the external toolchain url in a variable
 to make the long lines shorter and more readable.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-13 22:26:55 +01:00
Peter Korsgaard
7cfceeb91e utils/scanpypi: correctly handle license dirs in subdirs for .hash files
create_hash_file() used basename(licensefile) when it writes the entry for
the license file in the .hash, which is obviously not correct when license
file is locate in a sub directory.

Instead copy the logic from __create_mk_license() to strip the directory
prefix from the absolute filename instead.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-09 22:23:35 +01:00
Peter Korsgaard
92b8bd0879 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-05 21:04:14 +01:00
Peter Korsgaard
7cf95a8a78 utils/scanpypi: fix 'downloas' typo in error message
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-01 17:14:01 +01:00
Yegor Yefremov
c46f72b61e scanpypi: fix licence detection handling for unknown licences
Check for match object not being None.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-02-26 20:55:19 +01:00
Yegor Yefremov
cfafcfa9af scanpypi: fix Py2/3 conversion leftover
Use urlparse from six package.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-02-26 20:54:00 +01:00
Yegor Yefremov
3a0c20c530 scanpypi: add support for Python3
The script was changed via modernize utility. The only manual
made part was the handling of StringIO.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-02-25 22:21:54 +01:00
Arnout Vandecappelle (Essensium/Mind)
29d5b79b73 utils/genrandconfig: check core-dependencies
When generating a random configuration, also check the host
dependencies for that configuration and error out if it fails.

This is mainly intended for the autobuilders, so they don't try to
build a configuration for which the necessary host packages are not
installed.

Instead of erroring out, we could also generate a different random
config. However:

- When called for the autobuilders, the autobuild-run script will
  anyway just attempt another build. The only difference is that a
  warning will be printed in this autobuild log now. This warning
  is in fact quite useful to hint that the autobuilder could benefit
  from intalling this additional host package.

- When called manually, the user may in fact want to install the
  missing host package and continue with the saved random config.

Fixes:

http://autobuild.buildroot.net/results/d7c/d7c12e873a71fef1e0b5e19cfad45024050614e1
http://autobuild.buildroot.net/results/a0a/a0a105c1ec2d0517112b02932a85831d4b597b9f
http://autobuild.buildroot.net/results/96c/96cc6cedc64c777e993671204809599a1498472a

and many others...

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-02-08 22:20:08 +01:00
Ricardo Martincoski
2455e5a091 scanpypi: fix code style
Fix these warnings:
E101 indentation contains mixed spaces and tabs
E128 continuation line under-indented for visual indent
E231 missing whitespace after ','
E261 at least two spaces before inline comment
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
W191 indentation contains tabs

Ignore these warnings:
E402 module level import not at top of file

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
49ffceef57 get-developers: fix code style
Fix these warnings:
E202 whitespace before ']'
E203 whitespace before ':'
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
E711 comparison to None should be 'if cond is None:'
E741 ambiguous variable name 'l'
F401 'sys' imported but unused
W391 blank line at end of file

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
deb31a979a check-package: fix code style
Ignore these warnings:
F401 'lib.ConsecutiveEmptyLines' imported but unused

And remove comments that are not needed anymore.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Yegor Yefremov
09ec6d7f64 scanpypi: ignore empty elements in package requirements
Depending on how setup.py reads requirements files empty elements can occur.
This patch takes care, that such elements will be ignored and don't crash
the scanpypi script.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-01-26 14:43:28 +01:00
Yegor Yefremov
d2e29fccdf scanpypi: get rid of commented lines and also strip the package strings
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-18 22:11:09 +01:00
Jerzy Grzegorek
2f5421cb2c utils/checkpackagelib: add function to check of the default package source variable
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-15 23:27:02 +01:00
Yegor Yefremov
13d94669b9 scanpypi: add support for licence files checksums
Store the list of detected licence files in the main object and
automatically add their sha256 checksums when creating *.hash file.

Bonus: fix wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-15 22:12:47 +01:00
Yegor Yefremov
d2ac1ec6f4 scanpypi: get license names from SPDX database
Use spdx_lookup package to compare packages' license file texts
with SPDX database.

This feature is optional.

Bonus: fix wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-12 22:23:50 +01:00
Yann E. MORIN
17cd914917 utils/test-pkg: always run a global legal-info
Instead of limiting it to the package under test, we run it globally.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-08 09:16:50 +01:00
Alexey Roslyakov
5d2c69dad5 utils/scanpypi: increase error message verbosity
When package installation fails it is good to know what happened.

Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-12-18 15:18:40 +01:00
Ricardo Martincoski
29d9f2f014 check-package: avoid false warning of useless flag
Just AUTORECONF = NO is redundant.
Just HOST_AUTORECONF = NO is redundant.
But the combination of AUTORECONF = YES + HOST_AUTORECONF = NO is valid.

So basically for all variables that have inheritance between target and
host, having the host variant of the variable set the variable value
back to its default is correct if the target variable is set.

Instead of increasing complexity of the script to fully detect this
case, ignore the host flag set to its default value as it can be
overriding a non-default value inherited from the equivalent target
flag.

Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reported-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-12-02 14:51:27 +01:00
Thomas Petazzoni
9272eb4684 test-pkg: filter empty lines and comments in CSV file
In preparation for the addition of comments in the CSV file listing
toolchain configurations, we filter out such lines when reading the
CSV file in utils/test-pkg.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-11-27 23:24:49 +01:00
Thomas Petazzoni
c7abd1ddfb utils/genrandconfig: filter empty lines and comments in CSV file
In preparation for the addition of comments in the CSV file listing
toolchain configurations, we filter out such lines when reading the
CSV file in utils/genrandconfig.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-11-27 23:20:04 +01:00
Marcus Folkesson
e5b2fc8c7d utils/diffconfig: add diffconfig utility
Diffconfig is a simple utility for comparing two configuration files.
See usage in the script for more info.

Borrowed from the Linux kernel source code and adapted to Buildroot.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-11-25 13:46:41 +01:00
Francois Perrad
5338f8635d perl: bump to version 5.26.1
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Tested-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-09-28 22:04:48 +02:00
Marcus Folkesson
f2c5c7a263 utils/config: new script to manipulate .config files on the command line
Default prefix is set to `BR2_` but may be overidden by setting
BR2_PREFIX.

Example usage:

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --enable GNUPG

Check state of config option `BR2_PACKAGE_GNUPG`:
./utils/config --package --state GNUPG
y

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --disable GNUPG

Set `BR2_TARGET_GENERIC_ISSUE` to "Welcome":
./utils/config --set-str  TARGET_GENERIC_ISSUE "Welcome"

Copied from the Linux kernel (4.13-rc6) source code and adapted to
Buildroot.
Thanks to Andi Kleen who is the original author of this script.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
[Arnout: merge the two tr invications]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-09-27 22:45:35 +02:00
Francois Perrad
c92612b4b5 scancpan: catch exception when MANIFEST is missing
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-21 23:41:07 +02:00
Arnout Vandecappelle
009ae96c6f get-developers: add option to get developers for files
The DEVELOPERS file lists file patterns, but the get-developers script
only supports getting packages and CPU architectures. Some files are
neither of those, e.g. defconfigs, support files, package infras, ...

Add a '-f' option that allows to give a list of files for which to get
the developer(s).

Note that the DEVELOPERS file contains directories as well as files,
and the directories are not expanded into individual files. Therefore,
we have to use f.startswith(devfile) to match a directory. This assumes
that the directory name ends with /, otherwise 'package/python' would
also match 'package/python-foo'. Since we make sure that directories do
end with / in the DEVELOPERS file, and since false positives are not
much of an issue in this case, this isn't a problem.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-14 22:39:57 +02:00
Arnout Vandecappelle
a074e63449 utils/readme.txt: add documentation of genrandconfig
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-29 15:41:09 +02:00
Arnout Vandecappelle
ed59f81a3c utils/test-pkg: add option to use an alternate toolchains CSV file
When testing a package with test-pkg, it may be useful to override the
set of toolchains used. For example:
- to test with toolchains used in your company;
- to test against a subset that is known to be problematic;
- to use only toolchains you already have available locally when you
  have no network access.

Add an option to use an alternate CSV file containing the config
fragments of toolchains to try.

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-29 15:40:55 +02:00
Arnout Vandecappelle
de46cc9be0 test-pkg: use merge_config.sh to merge the fragments
It is supposedly more robust than just concatenating.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:07:03 +02:00
Arnout Vandecappelle
84929a53a4 genrandconfig: get configs from in-tree toolchain-configs.csv
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.

The --toolchains-url option is renamed to --toolchains-csv.

The paths in the toolchains_csv file should be either absolute, or
relative to buildrootdir.

After this change, the script should be called from autobuild-run as:

    subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
                     "-o", outputdir, "-b", srcdir,
                     "--toolchains-csv", kwargs['toolchains_csv']],
                    stdout=devnull, stderr=log)

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:05:29 +02:00
Arnout Vandecappelle
e7b9afa70a test-pkg: get configs from in-tree toolchain-configs.csv
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:03:28 +02:00
Arnout Vandecappelle
8cae6ac3e3 genrandconfig: use minimal.config
This has a number of side-effects which must be handled.

The lines in minimal.config may be overridden by the random lines added
by amending the configuration, so is_toolchain_usable() shouldn't take
those into account, or indeed the random lines added. Therefore, make
a copy of the config before appending minimal.config and the random
lines. While we're at it, rename the variable to the more appropriate
toolchainconfig.

minimal.config sets BR2_INIT_NONE=y, but we really also want to test
with BR2_INIT_BUSYBOX=y. Therefore, add a random line to use the
busybox init system. We set its probability rather high. The
probabilities of systemd and eudev are increased since they're now
in the else branch of BR2_INIT_BUSYBOX, which halves the probability
that we even get there.

We now also generate configurations without busybox. Previously,
busybox was almost always selected due to BR2_INIT_BUSYBOX=y. Only if
systemd is selected there was a chance to build without busybox.

We now set BR2_SYSTEM_BIN_SH_NONE=y, the other /bin/sh options are
never tested. However, this is not really something that is relevant
to test in the autobuilders.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:01:08 +02:00
Arnout Vandecappelle
1e70470545 genrandconfig: fix the case when outputdir is 'output'
When outputdir is 'output' (the default), genrandconfig didn't work
correctly because it expects the configfile in outputdir, while
Buildroot puts it in the buildroot directory.

Fix this by explicitly checking if outputdir == buildrootdir/output.
Because abspath is used for both paths, string comparison works
reliably.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:50:20 +02:00
Arnout Vandecappelle
5638b10c22 genrandconfig: calculate configfile only once
The path to the .config file is calculated in several places - replace
it with a single calculation, and pass configfile as an argument
to is_toolchain_usable and fixup_config. These functions also don't
need outputdir any more.

This makes it easier to fix the case when configfile is not in
outputdir.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:50:07 +02:00
Arnout Vandecappelle
b8288a5f43 genrandconfig: pass outputdir and buildrootdir as arguments
The --instance argument is just an artifact of genrandconfig's
history as part of autobuild-run. It is much more logical to pass
the output directory and the buildroot directory as arguments, with
sane defaults.

This also allows us to remove the hack of creating a symlink in the
instance directory if it doesn't exist yet.

Note that the default outputdir 'output' doesn't work yet, because in
that case Buildroot will put the config file in the buildroot directory
instead of the output directory. This will be fixed in a follow-up
patch.

After this change, the script should be called from autobuild-run as:

    subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
                     "-o", outputdir, "-b", srcdir,
                     "--toolchains-url", kwargs['toolchains_url']],
                    stdout=devnull, stderr=log)

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:44:19 +02:00
Arnout Vandecappelle
9633b6ddd2 genrandconfig: calculate buildrootdir in __main__
This prepares for passing buildrootdir as an argument.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:44:07 +02:00
Arnout Vandecappelle
2dc209be36 genrandconfig: calculate outputdir in __main__
This prepares for passing outputdir as an argument.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:43:50 +02:00