Commit Graph

180 Commits

Author SHA1 Message Date
Francois Perrad
5338f8635d perl: bump to version 5.26.1
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Tested-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-09-28 22:04:48 +02:00
Marcus Folkesson
f2c5c7a263 utils/config: new script to manipulate .config files on the command line
Default prefix is set to `BR2_` but may be overidden by setting
BR2_PREFIX.

Example usage:

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --enable GNUPG

Check state of config option `BR2_PACKAGE_GNUPG`:
./utils/config --package --state GNUPG
y

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --disable GNUPG

Set `BR2_TARGET_GENERIC_ISSUE` to "Welcome":
./utils/config --set-str  TARGET_GENERIC_ISSUE "Welcome"

Copied from the Linux kernel (4.13-rc6) source code and adapted to
Buildroot.
Thanks to Andi Kleen who is the original author of this script.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
[Arnout: merge the two tr invications]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-09-27 22:45:35 +02:00
Francois Perrad
c92612b4b5 scancpan: catch exception when MANIFEST is missing
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-21 23:41:07 +02:00
Arnout Vandecappelle
009ae96c6f get-developers: add option to get developers for files
The DEVELOPERS file lists file patterns, but the get-developers script
only supports getting packages and CPU architectures. Some files are
neither of those, e.g. defconfigs, support files, package infras, ...

Add a '-f' option that allows to give a list of files for which to get
the developer(s).

Note that the DEVELOPERS file contains directories as well as files,
and the directories are not expanded into individual files. Therefore,
we have to use f.startswith(devfile) to match a directory. This assumes
that the directory name ends with /, otherwise 'package/python' would
also match 'package/python-foo'. Since we make sure that directories do
end with / in the DEVELOPERS file, and since false positives are not
much of an issue in this case, this isn't a problem.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-08-14 22:39:57 +02:00
Arnout Vandecappelle
a074e63449 utils/readme.txt: add documentation of genrandconfig
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-29 15:41:09 +02:00
Arnout Vandecappelle
ed59f81a3c utils/test-pkg: add option to use an alternate toolchains CSV file
When testing a package with test-pkg, it may be useful to override the
set of toolchains used. For example:
- to test with toolchains used in your company;
- to test against a subset that is known to be problematic;
- to use only toolchains you already have available locally when you
  have no network access.

Add an option to use an alternate CSV file containing the config
fragments of toolchains to try.

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-29 15:40:55 +02:00
Arnout Vandecappelle
de46cc9be0 test-pkg: use merge_config.sh to merge the fragments
It is supposedly more robust than just concatenating.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:07:03 +02:00
Arnout Vandecappelle
84929a53a4 genrandconfig: get configs from in-tree toolchain-configs.csv
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.

The --toolchains-url option is renamed to --toolchains-csv.

The paths in the toolchains_csv file should be either absolute, or
relative to buildrootdir.

After this change, the script should be called from autobuild-run as:

    subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
                     "-o", outputdir, "-b", srcdir,
                     "--toolchains-csv", kwargs['toolchains_csv']],
                    stdout=devnull, stderr=log)

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:05:29 +02:00
Arnout Vandecappelle
e7b9afa70a test-pkg: get configs from in-tree toolchain-configs.csv
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:03:28 +02:00
Arnout Vandecappelle
8cae6ac3e3 genrandconfig: use minimal.config
This has a number of side-effects which must be handled.

The lines in minimal.config may be overridden by the random lines added
by amending the configuration, so is_toolchain_usable() shouldn't take
those into account, or indeed the random lines added. Therefore, make
a copy of the config before appending minimal.config and the random
lines. While we're at it, rename the variable to the more appropriate
toolchainconfig.

minimal.config sets BR2_INIT_NONE=y, but we really also want to test
with BR2_INIT_BUSYBOX=y. Therefore, add a random line to use the
busybox init system. We set its probability rather high. The
probabilities of systemd and eudev are increased since they're now
in the else branch of BR2_INIT_BUSYBOX, which halves the probability
that we even get there.

We now also generate configurations without busybox. Previously,
busybox was almost always selected due to BR2_INIT_BUSYBOX=y. Only if
systemd is selected there was a chance to build without busybox.

We now set BR2_SYSTEM_BIN_SH_NONE=y, the other /bin/sh options are
never tested. However, this is not really something that is relevant
to test in the autobuilders.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 23:01:08 +02:00
Arnout Vandecappelle
1e70470545 genrandconfig: fix the case when outputdir is 'output'
When outputdir is 'output' (the default), genrandconfig didn't work
correctly because it expects the configfile in outputdir, while
Buildroot puts it in the buildroot directory.

Fix this by explicitly checking if outputdir == buildrootdir/output.
Because abspath is used for both paths, string comparison works
reliably.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:50:20 +02:00
Arnout Vandecappelle
5638b10c22 genrandconfig: calculate configfile only once
The path to the .config file is calculated in several places - replace
it with a single calculation, and pass configfile as an argument
to is_toolchain_usable and fixup_config. These functions also don't
need outputdir any more.

This makes it easier to fix the case when configfile is not in
outputdir.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:50:07 +02:00
Arnout Vandecappelle
b8288a5f43 genrandconfig: pass outputdir and buildrootdir as arguments
The --instance argument is just an artifact of genrandconfig's
history as part of autobuild-run. It is much more logical to pass
the output directory and the buildroot directory as arguments, with
sane defaults.

This also allows us to remove the hack of creating a symlink in the
instance directory if it doesn't exist yet.

Note that the default outputdir 'output' doesn't work yet, because in
that case Buildroot will put the config file in the buildroot directory
instead of the output directory. This will be fixed in a follow-up
patch.

After this change, the script should be called from autobuild-run as:

    subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
                     "-o", outputdir, "-b", srcdir,
                     "--toolchains-url", kwargs['toolchains_url']],
                    stdout=devnull, stderr=log)

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:44:19 +02:00
Arnout Vandecappelle
9633b6ddd2 genrandconfig: calculate buildrootdir in __main__
This prepares for passing buildrootdir as an argument.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:44:07 +02:00
Arnout Vandecappelle
2dc209be36 genrandconfig: calculate outputdir in __main__
This prepares for passing outputdir as an argument.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:43:50 +02:00
Arnout Vandecappelle
d7b05d5b7d genrandconfig: verbose output and use stderr
The output of genrandconfig is currently very terse, which is annoying
for debugging the script or generally seeing what is going on. Also the
timing information added by log_write isn't very useful when the script
is used stand-alone.

In the new setup, (verbose) output goes to stdout and error output goes
to stderr. Also the "INFO: generate the configuration" message is
eliminated - it should go in the autobuild-run script.

We also add an explicit message when a toolchain can't be used after
the first defconfig, otherwise autobuild-run will just silently
restart.

Note that, since the output of make is no longer redirected to
/dev/null, we get one more message on stderr that will be recorded in
the autobuilder's log file: KCONFIG_SEED=0xXXXXXXXX.

This approach allows us to optimise the error handling to use
exceptions, where appropriate, which can be caught at the top level and
converted to an error message to stderr. This, in turn, allows us to use
subprocess.check_call, which eliminates a lot of conditions.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:43:30 +02:00
Arnout Vandecappelle
e8c6d52c89 genrandconfig: move instantiation of SystemInfo down
The SystemInfo class is instantiated globally and passed down to all
functions, but it is really only used in fixup_config. So instead,
instantiate it there.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:42:59 +02:00
Arnout Vandecappelle
22978c7399 genrandconfig: replace kwargs with explicit arguments
kwargs is a left-over from the use of docopt, it's better to use
argparse's Namespace object directly.

In addition, most functions use just one or two fields of args, so
these can just as well be passed directly as arguments to the function.
Particularly for outputdir it doesn't make sense to reconstruct it all
the time.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:42:45 +02:00
Arnout Vandecappelle
30f7fec0a8 genrandconfig: fix (some) pep8 warnings
Warnings fixed:
E731 do not assign a lambda expression, use a def
 -> urlopen_closing is defined with a def. urlopen is not used
    elsewhere so inlined.
E302 expected 2 blank lines
E501 line too long
 -> long lines due to a long string are NOT split
E701 multiple statements on one line (colon)
E722 do not use bare except'
 -> use "except Exception", so KeyInterrupt and SystemExit are still
    passed. We never intended to catch those.
E741 ambiguous variable name 'l'
 -> variable name is replaced with the much more descriptive
    toolchains_csv
E271 multiple spaces after keyword
E231 missing whitespace after ','

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:42:15 +02:00
Arnout Vandecappelle
59419cdac1 genrandconfig: use subprocess.check_output instead of Popen
Popen is more complicated and more difficult to understand.

check_output raises an exception if the exit code is non-zero, but
that's probably what we want if ldd can't be executed.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:41:57 +02:00
Arnout Vandecappelle
86ca7a04a8 utils/genrandconfig: new script
This script will be used by the autobuild-run script to generate the
configuration to test. It is put in the utils directory because it can
also be called directly to allow users to test things.

For now, it is a direct copy of the relevant functions from the
autobuild-run script. The only changes are:
- unneeded import statements are removed;
- code/decode wrappers are limited to decode_byte_list;
- __main__ handling is added.

For now, the only supported arguments are the ones needed for
autobuild-run.  Follow-up patches will refactor things and also change
the way the script is called. In this version, it can be called from the
autobuild-run script as:

  subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
                   "-i", str(kwargs['instance']),
                   "--toolchains-url", kwargs['toolchains_url']],
                  stdout=log, stderr=log)

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-25 22:41:35 +02:00
Arnout Vandecappelle
c23d0c8633 support/test-pkg: move minimal.config into a separate file
This minimal configuration is also very useful outside test-pkg. In
addition, it will simplify the config merge in a later patch.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-21 09:00:40 +02:00
Alexey Roslyakov
caa3760014 utils/scanpypi: include LICENSE.RST to supported license files
Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-20 22:28:00 +02:00
Alexey Roslyakov
c2877a2a8b utils/scanpypi: don't pass any arguments to main()
'if __name__ == "__main__"' idiom typically calls main function that
doesn't take any arguments in most cases. We shouldn't pass any tuple to
it.
I've tested the script with python-idna-2.5 and now it works with this
little change.

Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-20 22:27:58 +02:00
Yann E. MORIN
c8d8f74a93 utils/brmake: print the error code of the build
Since the stdout and stderr streams are redirected, it is not
immediately obvious when a build failed, even though brmake really exits
with the same error code as make did.

When there is an error, print the exit code after the elapsed time.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 00:15:05 +02:00
Yann E. MORIN
efb61ae07b support/check-package: don't check filenames of hashes
Currently, we check that the filenames in hash lists do not contain
a slash '/' character, because all we are checking so far are the
downloaded archives, and we explicitly need the filename to not contain
a directory component at all.

However, we're soon to also check the hashes of the license files in
packages sources, and those license files may be at any arbitrary
directory-depth in the packages source tree.

[Peter: Remove reference to files with same basename]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Rahul Bedarkar <rahulbedarkar89@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-07-03 17:33:22 +02:00
Thomas De Schampheleire
6ff0b344af utils/readme.txt: fix typos for size-stats-compare
A few typos sneaked into the new utils readme...

Additionally, I formulated the list of use cases a bit differently.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-02 00:35:37 +02:00
Arnout Vandecappelle
d01452ce6e get-developers: fix handling of syntax error
When getdeveloperlib.parse_developers doesn't find anything, the script
tries sys.exit(1) but sys is not imported.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 19:43:51 +02:00
Arnout Vandecappelle
9c2428616e scancpan: update old reference to support/scripts
scancpan's help text refers to its location, so update it after the
move to utils/.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 19:43:48 +02:00
Thomas Petazzoni
7ca9fc3170 tools: rename to 'utils'
After some discussion, we found out that "tools" has the four first
letters identical to the "toolchain" subfolder, which makes it a bit
unpractical with tab-completion. So, this commit renames "tools" to
"utils", which is more tab-completion-friendly.

This has been discussed with Arnout and Yann.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-01 18:07:00 +02:00