Commit Graph

326 Commits

Author SHA1 Message Date
Erico Nunes
2d9566b7b7 boot/grub2: add option to install tools
Add an option to install grub2 support tools to the target.

In the context of Buildroot, some useful target tools provided are
grub2-editenv, grub2-reboot, which provide means to manage the grub2,
environment, boot order, and others.

Signed-off-by: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-26 20:59:33 +01:00
Thomas Petazzoni
eeb6341410 support/scripts/check-host-rpath: split condition on two statements
Inside the check_elf_has_rpath(), we check if the host binary has a
correct RPATH, which should be either an absolute path to
$(HOST_DIR)/lib, or a relative path using $ORIGIN. Those two
conditions are checked in a single statements, but as we are going to
add a third condition, let's split this up a bit:

 - If we have a RPATH to $(HOST_DIR)/lib -> we're good, return 0
 - If we have a RPATH to $ORIGIN/../lib -> we're good, return 0
 - Otherwise, we will exit the loop, and return 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:08:57 +01:00
Serj Kalichev
c77cd17082 support/scripts/mkmakefile: make wrapper silent by default
Suppose we use Makefile wrapper and build some project out of
buildroot tree (O=...). A command like "make
busybox-all-external-deps" will output the string "uname 022 && make
..." to stdout before the usefull information. It pollutes stdout. At
the same time if we use the same command in the buildroot source-tree
then we don't get the additional output. This patch makes wrapper
silent by default. People who prefer to see more verbose output can
use V=1.

Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-02 21:23:02 +01:00
Thomas Petazzoni
b2c24f584d package/pkg-generic.mk: increase precision of timestamps
Currently, the timestamps that we keep in build-time.log use a
second-level precision. However, as we are going to introduce a new
type of graph to draw the time line of a build, this precision is
going to be insufficient, as a number of steps are so short that they
are not even one second long, and generally the rounding to the second
gives a not so great looking graph.

Therefore, we add to the timestamps the nanoseconds using the %N date
specifier. A milli-second precision would have been sufficient, but %N
is all what date(1) provides at the sub-second level.

Since this is changing the format of the build-time.log file, this
commit adjusts the support/scripts/graph-build-time script
accordingly, to account for the floating point numbers that we have as
timestamps.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 21:30:51 +02:00
Matt Weber
3b657e264c support/scripts/pkg-stats: URL check using threads
Adds a pool of worker threads to accelerate connection testing.

~7.5MB and 2% CPU per thread on a Intel i5-3230M CPU @ 2.60GHz.

Runtime is ~3min in parallel vs ~15min.

CC: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-09 10:11:47 +02:00
Matt Weber
b4da3b2603 support/scripts/pkg-stats: URL checking support
- Adds support to check if a package has a URL and if that URL
  is valid by doing a header request.
- Reports this information as part of the generated html output

The URL data is currently gathered from the URL string provided
in the Kconfig help sections for each package.

This check helps ensure the URLs are valid and can be used
for other scripting purposes as the product's home site/URL.
CPE XML generation is an example of a case that could use this
product URL as part of an automated update generation script.

CC: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-09 10:05:46 +02:00
Johan Oudinet
497f7134fc support/scripts/mkusers: preserve group members
When the function add_one_group is called on an existing group,
make sure the members of this group are not removed in the process of
deleting then re-adding the group.

Signed-off-by: Johan Oudinet <johan.oudinet@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: add curly braces when referencing ${members}, as suggested by
Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-12 21:22:25 +02:00
Matt Weber
eb04235783 support/scripts/pkg-stats: use parenthesis for print
Use Python 3 style print calls, in order to make pkg-stats Python 3
compliant.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-19 13:48:21 +02:00
Thomas Petazzoni
ea7f7aaa6d support/scripts: drop xorg-release script
This script causes a large number of flake8 warnings, is rarely used
(but even never used), and is going to be replaced at some point by
the improved pkg-stats that will give details about the upstream
version available for all packages, not just X.org packages.

Therefore, let's drop the xorg-release script in order to silence all
those flake8 warnings:

support/scripts/xorg-release:36:1: E302 expected 2 blank lines, found 1
support/scripts/xorg-release:58:27: E201 whitespace after '{'
support/scripts/xorg-release:58:44: E203 whitespace before ':'
support/scripts/xorg-release:58:54: E202 whitespace before '}'
support/scripts/xorg-release:63:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/xorg-release:64:15: E261 at least two spaces before inline comment
support/scripts/xorg-release:67:32: E261 at least two spaces before inline comment
support/scripts/xorg-release:86:1: E302 expected 2 blank lines, found 1
support/scripts/xorg-release:95:1: E302 expected 2 blank lines, found 1
support/scripts/xorg-release:107:1: E302 expected 2 blank lines, found 1
support/scripts/xorg-release:115:20: W601 .has_key() is deprecated, use 'in'
support/scripts/xorg-release:123:34: E201 whitespace after '{'
support/scripts/xorg-release:124:46: E203 whitespace before ':'
support/scripts/xorg-release:124:50: E202 whitespace before '}'
support/scripts/xorg-release:127:1: E302 expected 2 blank lines, found 1
support/scripts/xorg-release:141:15: W601 .has_key() is deprecated, use 'in'
support/scripts/xorg-release:146:21: W601 .has_key() is deprecated, use 'in'
support/scripts/xorg-release:176:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/xorg-release:180:1: W391 blank line at end of file

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-27 22:50:36 +02:00
Thomas Petazzoni
34d32e911f support/scripts/check-uniq-files: remove csv module import
Since commit 5563a1c6a4
("support/check-uniq-files: support weird locales and filenames"), the
'csv' Python module is no longer used by the check-uniq-files.

Due to this, flake8 complains with:

support/scripts/check-uniq-files:4:1: F401 'csv' imported but unused

Fix this by dropping the useless csv import.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-25 21:19:22 +02:00
Thomas Petazzoni
184cb52f6d support/scripts/fix-rpath: exclude /lib/firmware in the target
The /lib/firmware directory contains random firmware for various
devices. It happens that some of them might be or appear to be ELF
files, but they shouldn't be checked by fix-rpath. For example, one of
the Qualcomm VPU firmware file appears to be an ELF file, but patchelf
isn't happy about it:

$ ./output/host/bin/patchelf --print-rpath output/target/lib/firmware/qcom/venus-4.2/venus.b00
patchelf: patchelf.cc:387: void ElfFile<Elf_Ehdr, Elf_Phdr, Elf_Shdr, Elf_Addr, Elf_Off, Elf_Dyn, Elf_Sym>::parse() [with Elf_Ehdr = Elf32_Ehdr; Elf_Phdr = Elf32_Phdr; Elf_Shdr = Elf32_Shdr; Elf_Addr = unsigned int; Elf_Off = unsigned int; Elf_Dyn = Elf32_Dyn; Elf_Sym = Elf32_Sym]: Assertion `shstrtabIndex < shdrs.size()' failed.
Aborted (core dumped)

Even though patchelf definitely shouldn't crash, it anyway doesn't
make sense to check ELF files in /lib/firmware, so let's exclude this
directory from our check.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-12 21:15:49 +02:00
Thomas Petazzoni
f955a4eac0 support/scripts/pkg-stats: replace with new Python version
Rename pkg-stats-new to pkg-stats.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-04 22:05:33 +02:00
Thomas Petazzoni
338c7b5e62 support/scripts/pkg-stats-new: add current version information
This commit adds a new column in the HTML output containing the
current version of a package in Buildroot. As such, it isn't terribly
useful, but combined with the latest upstream version added in a
follow-up commit, it will become very useful.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-04 22:04:58 +02:00
Thomas Petazzoni
df8a39468b support/scripts/pkg-stats-new: add -n and -p options
This commit adds the following options to the pkg-stats-new script:

 -n, to specify a number of packages to parse instead of all packages

 -p, to specify a list of packages (comma-separated) to parse instead
     of all packages

These options are basically only useful when debugging/developing
this script, but they are very useful, because the script is rather
slow to run completely with all 2000+ packages, especially once
upstream versions will be fetched from release-monitoring.org.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-04 22:04:44 +02:00
Thomas Petazzoni
ef63a330b4 support/scripts/pkg-stats-new: rewrite in Python
This commit adds a new version of the pkg-stats script, rewritten in
Python. It is for now implemented in a separate file called,
pkg-stats-new, in order to make the diff easily readable. A future
commit will rename it to pkg-stats.

Compared to the existing shell-based pkg-stats script, the
functionality and output is basically the same. The main difference is
that the output no longer goes to stdout, but to the file passed as
argument using the -o option. This allows stdout to be used for more
debugging related information.

The way the script works is that a first function get_pkglist()
returns a list of Package objects. Then, the function
package_init_make_info() uses 'make printvars' to gather information
about all packages, stored as class variables in the Package
class. Then, we iterate over all packages, and use various methods of
the Package class to retrieve all details about the package:
infrastructure, presence of hash file, presence of license
information, etc.

calculate_stats() then calculates global statistics (how packages have
license information, how packages have a hash file, etc.). Finally,
dump_html() produces the HTML output, using a number of sub-functions.

One improvement over the shell-based version is that we can use
regexps to exclude some .mk files. Thanks to this, we can exclude all
linux-ext-*.mk files, avoiding incorrect matches.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
2018-04-04 22:01:55 +02:00
George Redivo
3146ba7633 support/scripts/graph-depends: add --flat-list option
graph-depends currently spits out a graph in .dot format. However, as
part of the upcoming introduction of <pkg>-show-recursive-depends and
<pkg>-show-recursive-rdepends, we need graph-depends to be able to
display a flat list.

Signed-off-by: George Redivo <george.redivo@datacom.ind.br>
[Thomas:
 - Rebase on top of graph-depends changes
 - Do not display the package name itself in the list, only its
   dependencies (or reverse dependencies)
 - Display the result on a single line, instead of one package per
   line, in order to match what <pkg>-show-depends and
   <pkg>-show-rdepends are doing today.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 22:25:57 +02:00
Peter Korsgaard
dcfcb777f4 support/scripts/graph-depends: add --quiet option
This will be useful for the upcoming recursive show-depends and
show-rdepends features.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 22:25:57 +02:00
Peter Korsgaard
3543d17441 support/scripts/graph-depends: use the standard python logging module
Instead of hardcoded sys.stderr.write() calls.  No functional change, but
allows us to easily implement a quiet option.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 22:25:57 +02:00
Thomas Petazzoni
3cd92ac711 support/scripts/graph-depends: use colors instead of colours
The graph-depends was not very consistent in colors vs. colours: some
parts were using colours, some parts were using colors.

Let's settle on the US spelling, colors.

This change the user-visble option --colours to --colors, but it is
unlikely that a lot of users customize the colors through
BR2_GRAPH_DEPS_OPTS, so this user interface change is considered
reasonable.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 20:01:38 +02:00
Thomas Petazzoni
f179394621 support/scripts/graph-depends: remove global code and most global variables
The graph-depends script had no main() function, and the main code was
actually spread between the function definitions, which was a real
mess.

This commit moves the global code into a main() function, which allows
to more easily follow the flow of the script. The argument parsing
code is moved into a parse_args() function.

Most of the global variables are removed, and are instead passed as
argument when appropriate. This has the side-effect that the
print_pkg_deps() function takes a lot of argument, but this is
considered better than tons of global variables.

The global variables that are removed are: max_depth, transitive,
mode, root_colour, target_colour, host_colour, outfile, dict_deps,
dict_version, stop_list, exclude_list, arrow_dir.

The root_colour/target_colour/host_colour variables are entirely
removed, and instead a single colours array is passed, and it's the
function using the colors that actually uses the different entries in
the array.

The way the print_attrs() function determines if we're display the
root node is not is changed. Instead of relying on the package name
and the mode (which requires passing the root package name, and the
mode), it relies on the depth: when the depth is 0, we're at the root
node.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-04-01 20:01:38 +02:00
Yann E. MORIN
5563a1c6a4 support/check-uniq-files: support weird locales and filenames
Currently, when a filename contains characters not representable in the
user's locale, we fail hard, especially when the host python is python3.

This is because python2 and python3 handle encoding/decoding strings
differently, with python3 presumable doing the right thing, but it
breaks on some systems, while python2 presumable does the wrong thing,
but it works everywhere. (Just joking, obviously...)

Part of the issue being that the csv reader in python2 is broken with
UTF8.

We fix the issue by ditching the csv reader, and simply read the file in
binary mode, manually partitioning the lines on the first comma.

Then, we use the binary-encoded (really, un-encoded) package names and
filenames as values and keys, respectively.

Finally, for each filename or package we need to print, we try to decode
them with the defaults for the user settings, but catch any decoding
exception and fall back to dumping the raw, binary values. Which codec
is used by default differs between Python version, but in all cases
something sane is printed at least.

Thanks a lot to Arnout for the live help doing this patch. :-)

Reported-by: Jaap Crezee <jaap@jcz.nl>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Jaap Crezee <jaap@jcz.nl>
[Arnout: commit log improvement]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-03-31 15:32:00 +02:00
Yann E. MORIN
1b9a57442d support/check-bin-arch: exclude kernel modules for merged /usr
When using a merged /usr, the kernel module path is really
/usr/lib/modules, as /lib is a symlink to usr/lib .

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-07 23:03:52 +01:00
Yann E. MORIN
01d90f0d09 spport/check-bin-arch: accept arbitrary per-package ignore paths
Some packages (mostly, out-of-tree) may want to install binary blobs for
another architecture,  outside the locations we currently exclude, like
in /opt or whatever...

Add support in check-bin-arch to accept any arbitrary location, that
individual package can each request to excude from the check, when they
are installed.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-07 23:03:27 +01:00
Thomas Petazzoni
62fa5e17cb support/scripts/check-uniq-files: add indices in format string
Using {} in format strings is only supported in sufficiently recent
Python versions. Python 2.6 doesn't support this, and only format
strings with numbered arguments: {0}, {1}, etc.

Python 2.7:

$ python -c 'print("foo {}".format(12))'
foo 12
$ python -c 'print("foo {0}".format(12))'
foo 12

Python 2.6:

$ python -c 'print("foo {}".format(12))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: zero length field name in format
$ python -c 'print("foo {0}".format(12))'
foo 12

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-03 17:17:41 +01:00
Thomas De Schampheleire
78ec4bd9e6 check-bin-arch: skip /lib/modules to allow 32-bit userland on 64-bit arch
The script check-bin-arch fails as follows on a config for PowerPC e6500
(64-bit CPU) with BR2_ARCH="powerpc" (32-bit userland desired):

ERROR: architecture for "/lib/modules/..../lib/libcrc32c.ko"
            is "PowerPC64", should be "PowerPC"

This situation is perfectly acceptable: the kernel is 64-bit and so are its
modules, even though userland is 32-bit.

To keep check-bin-arch and its caller simple, just skip /lib/modules/
entirely, like is done for /lib/firmware and some others.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-02-25 23:16:05 +01:00
Ricardo Martincoski
493a86a7b7 size-stats: fix code style
Fix these warnings:
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
E713 test for membership should be 'not in'

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
253e016616 pycompile: fix code style
Fix these warnings:
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
cd04833926 graph-build-time: fix code style
Fix these warnings:
E201 whitespace after '['
E202 whitespace before ']'
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1

Ignore these warnings:
E402 module level import not at top of file

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
236bede631 check-uniq-files: fix code style
Fix these warnings:
E128 continuation line under-indented for visual indent
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Ricardo Martincoski
9ff44852c4 graph-depends: fix code style
Fix these warnings:
E122 continuation line missing indentation or outdented
E127 continuation line over-indented for visual indent
E128 continuation line under-indented for visual indent
E202 whitespace before ']'
E221 multiple spaces before operator
E225 missing whitespace around operator
E231 missing whitespace after ','
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
E502 the backslash is redundant between brackets
E713 test for membership should be 'not in'

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-29 23:14:24 +01:00
Thomas De Schampheleire
a3f37c53d5 support/scripts: sort pie charts
Make sure that the pie charts produced by 'graph-build' and 'graph-size'
targets are sorted on the size of each piece of the pie. Otherwise, making
visual analysis is difficult, as one needs to look at the legends of each
piece and do the sorting manually in their head.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2018-01-02 22:40:09 +01:00
Luca Ceresoli
6887ad7003 support/scripts/genimage.sh: show usage when invoked incorrectly
[Peter: use ${0}]
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-12-11 20:14:39 +01:00
Thomas Petazzoni
1c8dda3e43 Merge branch 'next'
This merges the next branch accumulated during the 2017.11 release
cycle back into the master branch.

A few conflicts had to be resolved:

 - In the DEVELOPERS file, because Fabrice Fontaine was added as a
   developer for libupnp in master, and for libupnp18 in
   next. Resolution is simple: add him for both.

 - linux/Config.in, because we updated the 4.13.x release used by
   default in master, while we moved to 4.14 in next. Resolution: use
   4.14.

 - package/libupnp/libupnp.hash: a hash for the license file was added
   in master, while the package was bumped into next. Resolution: keep
   the hash for the license file, and keep the hash for the newest
   version of libupnp.

 - package/linux-headers/Config.in.host: default version of the kernel
   headers for 4.13 was bumped to the latest 4.13.x in master, but was
   changed to 4.14 in next. Resolution: use 4.14.

 - package/samba4/: samba was bumped to 4.6.11 in master for security
   reasons, but was bumped to 4.7.3 in next. Resolution: keep 4.7.3.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-12-01 21:56:44 +01:00
Andrey Yurovsky
88af7d330d support/scripts/size-stats: avoid divide-by-zero
Some packages (ex: skeleton-init-systemd) have a zero size so we cannot
divide by the package size. In that case make their percent zero
explicitly and avoid a ZeroDivisionError exception.

Signed-off-by: Andrey Yurovsky <yurovsky@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-11-29 21:36:34 +01:00
Yann E. MORIN
1b8ad2d08e core: check files are not touched by more than one package
Currently, we do nothing about packages that touch the same file: given
a specific configuration, the result is reproducible (even though it
might not be what the user expected) because the build order is
guaranteed.

However, when we later introduce top-level parallel build, we will no
longer be able to guarantee a build order, by the mere way of it being
parallel. Reconciliating all those modified files will be impossible to
do automatically. The only way will be to refuse such situations.

As a preliminary step, introduce a helper script that detects files that
are being moified by two or more packages, and reports them and the
impacted packages, at the end of the build.

The list being reported at the end of the build will make it prominently
visible in autobuilder results, so we can assess the problem, if any.

Later on, calling that helper script can be done right after the package
installation step, to bail out early.

Thanks Arnout for the pythonist way to write default dictionaries! ;-)

Note: doing it in python rather than a shell script is impressively
faster: where the shell script takes ~1.2s on a minimalist build, the
python script only takes ~0.015s, that is about 80 times faster.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Peter Seiderer <ps.report@gmx.net>
[Thomas: rename script without .py extension.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-11-27 23:02:19 +01:00
Yann E. MORIN
c1cfee290e support/check-rpath: recognise PIE
We sanity-check the host executables that they have a correct RPATH
pointing to the host libraries.

This is currently done by looking for all files in $(HOST_DIR) that
match the 'ELF executable' pattern (a bit more complex, but that's
idea).

However, when an executable is built with -fPIE of -fpie, it no longer
appears to be an 'ELF executable', but it rather looks like an 'ELF
sheard object' (like if it were an library.

So, we miss those files.

It turns out that the problem is a real one, because quite a few
mainline distros, expecially those based on Debian for example, have
already switched to generating PIE code by default, and thus we miss on
a whole class of systems..

We fix that by simply looking if we can find an ELF interpreter in each
file. If we there is one, this is an ELF executable; if not, it may be
anything else: we don't care (not even about ELF libraries).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-09-27 00:13:55 +02:00
Thomas Petazzoni
85f11d08c3 support/scripts/apply-patches.sh: do not apply patches with renames
Patches with renames apply properly with patch >= 2.7, but not with
older patch versions. Since "git format-patch" by default generates
patches with renames, Buildroot developers often don't realize that
their patches will not apply properly on build machines that have
patch < 2.7. In order to prevent such a situation from happening
again, this commit adds some logic in apply-patches.sh to refuse
applying patches that contain renames.

Note that just searching for '^rename' is not sufficient, since the
patch commit message may contain the words "rename from" or "rename to"
as well. Therefore, the grep expression is made as accurate as possible,
checking both.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Arnout: spaces instead of tabs (suggested by Yann);
 extend commit message.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2017-09-19 22:51:25 +02:00
Thomas Petazzoni
cb0f3fa13f support/scripts/check-bin-arch: fix symbolic link check
Commit c96b8675ea
("support/scripts/check-bin-arch: ignore symbolic links") was bogus,
because it tested ${f}, which is the relative path of the file inside
${TARGET_DIR}, so we end up testing if ${f} on the system is a
symbolic link.

This commit fixes that by testing ${TARGET_DIR}/${f}.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-09-05 20:59:04 +02:00
Thomas Petazzoni
c96b8675ea support/scripts/check-bin-arch: ignore symbolic links
Since commit da32b49f00
("instrumentation: extend packages-file-list.txt with symlinks and
directories"), the packages-file-list.txt also contains symbolic
links. Therefore, check-bin-arch is now also checking symbolic links.

However, symbolic links in $(TARGET_DIR) can have absolute path as
targets, such as:

$ ls -l output/target/sbin/ifdown
lrwxrwxrwx 1 thomas thomas 10 Sep  3 15:55 output/target/sbin/ifdown -> /sbin/ifup

Therefore, we are now potentially checking a host binary, which
obviously makes check-bin-arch fail.

This commit changes check-bin-arch to ignore symbolic links. Indeed,
we have two cases:

 - The symbolic link really points to something that will in the
   rootfs (such as /sbin/ifup above). In this case, /sbin/ifup will be
   checked separately by check-bin-arch.

 - The symbolic link doesn't point to something that will be in the
   rootfs, and that is not a problem from the perspective of
   check-bin-arch, which checks the architecture of target binaries.

Fixes:

  http://autobuild.buildroot.net/results/16d384a0183d477646ac7692feb65f00dde7d068/
  (vim)

  http://autobuild.buildroot.net/results/50429c0f63a8befff9e20899327b9a8d754d99be/
  (ifupdown)

  http://autobuild.buildroot.net/results/1db65973e782bfa61abcbccd3501bfd235f77288/
  (gawk)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-09-04 16:01:51 +02:00
Wolfgang Grandegger
0bc81f528b support/scripts/fix-rpath: typo fix and unneeded "; " removed
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 14:38:43 +02:00
Wolfgang Grandegger
17dd0d49da support/scripts/fix-rpath: check if the patchelf utility is executable
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-22 14:35:55 +02:00
Samuel Martin
022b8adbd6 support/scripts: add fix-rpath script to sanitize the rpath
This commit introduces the script "fix-rpath" able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.
The RPATH fixup is done by the patchelf utility using the option
"--make-rpath-relative <root-directory>".

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-20 22:47:04 +02:00
Thomas Petazzoni
67409d9303 pkg-stats: add exception for nvidia-tegra23.mk
This file is not a package per-se, it includes other .mk files that
are packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-06 21:46:04 +02:00
Thomas Petazzoni
df3bf7bccd pkg-stats: take into account waf packages
The waf package infrastructure was not known by the pkg-stats script,
so let's add it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-06 21:45:41 +02:00
Ricardo Martincoski
2f762fb620 pkg-stats: allow to sort by column
With 2000+ packages it's not trivial to identify i.e.:
- all packages that don't have a hash file;
- all packages that have patches;
- all packages that have code style warnings;

User experience can be improved by dynamically sorting the resulting
table.

There is an open-source solution that does that in the client-side and
requires minimal changes to our script: sorttable.js.  The script is
MIT licensed as stated in its website.

Also add a hint to the user that the table can be sorted.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-06 21:38:43 +02:00
Arnout Vandecappelle
1d11d6314c check-host-rpath: no longer allow $(HOST_DIR)/usr
Now all packages have been updated to install things in $(HOST_DIR)/lib
instead of $(HOST_DIR)/usr/lib, there should no longer be any reason
to have $(HOST_DIR)/usr/lib in the RPATH, so we don't allow it any more.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Wolfgang Grandegger <wg@grandegger.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 16:54:12 +02:00
Arnout Vandecappelle
4db7d1f355 eclipse-register-toolchain: toolchain is no longer installed in $(HOST_DIR)/usr
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 16:06:27 +02:00
Arnout Vandecappelle
2295780a95 check-host-rpath: no longer check $(HOST_DIR)/usr/{bin, sbin}
Since $(HOST_DIR)/usr/{bin,sbin} are now symlinks to
$(HOST_DIR)/{bin,sbin}, it makes no sense to check them - they are
already covered.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 11:45:58 +02:00
Arnout Vandecappelle
4c790b8864 Move $(HOST_DIR)/usr/lib to $(HOST_DIR)/lib
This is a step towards eliminating $(HOST_DIR)/usr. It allows us to
convert all packages installing things into $(HOST_DIR)/usr/lib without
affecting the rest.

To allow compatibility with packages that still use $(HOST_DIR)/usr as
the prefix, create a symlink from usr/lib to ../lib.

Note that the symlink creation will break when $(HOST_DIR)/usr/lib
already exists as a directory, i.e. when rebuilding in an existing
output directory. This is necessary: if we don't break it now, the
following commits (which remove the usr part from various variables)
_will_ break it.

At the same time as creating this symlink, we also have to update the
check-host-rpath script to accept both $(HOST_DIR)/usr/lib and
$(HOST_DIR)/lib, because depending on how the package derives the
path, it may be different.

Since there are some dependency chains that involve $(STAGING_DIR),
$(STAGING_DIR) may in fact be created before $(HOST_DIR). Since
$(STAGING_DIR) is a subdirectory of $(HOST_DIR), it is possible that the
newly added rule for $(HOST_DIR) never triggers. To make sure that the
rule does trigger, add an order-only dependency from $(STAGING_DIR) to
$(HOST_DIR).

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 11:39:58 +02:00
Thomas De Schampheleire
ec019bcf64 setlocalversion: fix detection of hg revision for untagged versions
By default, cut prints the entire line if the specified delimiter is not
present at all:

    $ printf "foo bar" | cut -d' ' -f2
    bar
    $ printf "foobar" | cut -d' ' -f2
    foobar

In setlocalversion, cut is presented with the output of 'hg id' which has
the format:

    "<revision> <tags-if-any>"

If the current revision is not tagged, the output of 'hg id' does not
contain the delimiter (space), cut prints the entire string, and
setlocalversion thinks the version is the tag.
As setlocalversion does not print anything for tagged versions, there is no
output overall, and no correct indication of the mercurial revision.

Fix by passing the extra cut option '--only-delimited', which suppresses
output if no delimiter is found.

This problem likely went unnoticed for so long, because the tag 'tip' (i.e.
most recent revision of the branch) is treated specially: in this case the
mercurial revision _is_ printed, i.e. the situation is treated as
'untagged'.
The problem is only seen when you are _not_ at the most recent revision in
your branch.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-07-03 11:40:49 +02:00