Commit Graph

199 Commits

Author SHA1 Message Date
Thomas Petazzoni
34b1bdeba4 size-stats: fix bug when parsing file names with comma
The size-stats script fails when the usb_modeswitch_data is enabled,
because this package installs files that contain commas in their
name. However, the size-stats script also uses comma as a separator for
its CSV files, causing a "ValueError: too many values to unpack" in:

  pkg, fpath = l.split(",")

Fix this by splitting only the two fields that need to be split.

The bug was reported by Matthias <porto.rio@gmx.net>, who also suggested
a fix.

Fixes bug #9136.

Reported-by: Matthias <porto.rio@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-08-09 12:25:11 +02:00
Romain Naour
bd3c4d8c7f support: pkg-stats: add efl package in packages stats
Since efl update to 1.15 version, the efl package is a "real"
Buildroot package. It doesn't contain any subdirectories anymore.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-24 23:25:31 +02:00
Yegor Yefremov
28eae03f5e scanpypi: include LICENCE in the list of supported license files
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-07 11:48:52 +02:00
Yann E. MORIN
d5ae67b4f4 support/apply-patches: re-instate set -e
As reported by Sébastien Szymanski [1], the apply-patches script
doesn't stop if a tar command can't extract an archive.

Use "set -e" to exit immediately if a command return an error.

Be sure to ignore any expected error: when we check if a patch to be
applied has the same basename as an already applied patch, the grep
would fail when no such patch was already applied. We should not fail
in this case.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-06 14:47:30 +02:00
Thomas Petazzoni
75d8977ba8 Revert "apply-patches: catch unexpected failure"
This reverts commit 9cf1ad6cdb.

This breaks many packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-05 17:42:27 +02:00
Romain Naour
9cf1ad6cdb apply-patches: catch unexpected failure
As reported by Sébastien Szymanski [1], the apply-patches script
doesn't stop if a tar command can't extract an archive.

Use "set -e" to exit immediately if a command return an error.

[1] http://patchwork.ozlabs.org/patch/626196

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-05 17:36:57 +02:00
Francois Perrad
28a001fc7b scancpan: improve message when bad host perl version
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
[Thomas:
 - add comment in scancpan about the version dependency, suggested by
   Yann E. Morin.
 - add comment in perl.mk about the need to sync any version change with
   scancpan, also suggested by Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-04 17:41:54 +02:00
Yegor Yefremov
2f4bd0f339 scanpypi: change hash file comments
sha256 checksum will be computed locally either by scanpypi at package
creation or by hand by package updates. Define this checksum as
'computed locally' so that one doesn't need to change this comment by
package updates. Also put comments for both md5 and sha256 in one line.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-04 16:57:10 +02:00
Yegor Yefremov
f49295cc42 scanpypi: fix help text handling
Make sure a help text is terminated with a full stop.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-04 00:35:16 +02:00
Yegor Yefremov
8a64ade2f3 scanpypi: rework runtime dependency handling
In most cases Python's package dependencies found in setup.py are
runtime dependencies and hence don't need to be mentioned in *.mk
file.

Also add '# runtime' tag to select statements in Config.in.

__create_mk_requirements() itself is left for future uses (cffi backend
handling etc.).

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-07-01 00:03:54 +02:00
Yann E. MORIN
1924159814 support/apply-patches: bail-out on duplicate patch basenames
Patches we save can come from various locations:
  - bundled with Buildroot
  - downloaded
  - from one or more global-patch-dir

It is possible that two patches lying into different locations have the
same basename, like so (first is bundled, second is from an hypothetical
global-patch-dir):
    package/foo/0001-fix-Makefile.patch
    /path/to/my/patches/foo/0001-fix-Makefile.patch

In that case, when running legal-info, we'd save only the second patch,
overwriting the first. That would be problematic, because:

  - either the second patch depends on the first, and thus would no longer
    apply (this is easy to detect, though),

  - or the second patch does not depend on the first, and the compliance
    delivery will not be complete (this is much harder to detect).

We fix that by checking that no two patches have the same same basename.
If we find that the basename of the patch to be applied collides with
that of a previously applied patch, we error out and report the duplicate.

The unfortunate side-effect is that existing setups will now break in
that situation, but that's a minor, corner-case issue that is easily
fixed.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: adjust coding style, fix minor typos in the commit log.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-06-24 16:04:24 +02:00
Yann E. MORIN
a3165461bb core/apply-patches: store full path of applied patches
Currently, we only store the filename of the applied patches.

However, we are soon to want to install those patches in the legal-info
directory, so we'll have to know where those patches come from.

Instead of duplicating the logic to find the patches (bundled,
downloaded, from a global patch dir...), just store the full path to
each of those patches so we can retrieve them more easily later on.

Also always create the list-file, even if empty, so that we need not
test for its existence before reading it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
[Tested only with patches in the Buildroot sources]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: used $PWD instead of $(pwd), as suggested by Arnout.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-06-24 15:58:38 +02:00
Denis THULIN
1008ac4406 scanpypi: new utility
A utility for creating python package from the python package index.
It fetches packages info from http://pypi.python.org and generates
corresponding packages files.

Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
Tested-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
[Thomas: minor tweaks.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-06-11 16:13:31 +02:00
Peter Korsgaard
577021e81b Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-06-01 17:55:16 +02:00
Yegor Yefremov
549bbba67f python/python3: globalize *.pyc files compilation
Currently, each python package (be it the python interpreter package
itself or external python modules) is responsible for compiling its
.py into .pyc files. Unfortunately, this is not ideal as some packages
only install .py files without compiling them into .pyc files. In this
case, if the Buildroot configuration specifies to keep only the .pyc
files, the .py files are removed and lost.

To address this, this commit changes the logic by making the
compilation of .pyc files a global operation: the python interpreter
packages register a target finalize hook that is in charge of
compiling all installed .py files.

The *.pyc generation on a per package basis is disabled in the
python-package infrastructure by passing the "--no-compile" option to
setup.py.

The *.pyc generation for the Python interpreter internal modules is
disabled through --disable-pyc-build configure option.

A small helper script is used to perform the compilation, the purpose
of this script is to abort the compilation process if one of the .py
file cannot be compiled. It has been provided by Samuel Martin and
integrated into this commit.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas:
 - rework for python 3.5
 - integrate Samuel proposal that allows to detect compilation
   failures.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-26 22:07:35 +02:00
Thomas Petazzoni
8bf947e0ef support/scripts: fix graph-build-time help text
The graph-build-time help text currently looks like this:

usage: graph-build-time [-h] [--type GRAPH_TYPE] [--order GRAPH_ORDER]
                        [--alternate-colors] [--input OUTPUT] --output OUTPUT

Obviously, naming the parameter for --input as OUTPUT is not a very
good idea, so this commit fixes that to name it "INPUT", as expected.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-24 23:26:25 +02:00
Yann E. MORIN
88bad8f462 support/scripts: add helper to hardlink-or-copy
When preparing the legal-info, the source archives are copied in the
legal-info/ output directory. When the archives are big, it can take
quite a bit of time and unnecessarily uses disk space. When the
legal-info output directory is on the same filesystem as the BR2_DL_DIR,
we can easily reduce copy time and disk usage by just using hardlins
instead of copying. However, the BR2_DL_DIR may be on a different
filesystem, so we must fallback to copying in this case

Introduce a helper script that copies a source file into a destination
directory, by first attempting to hard-link, and falling back to a
plain copy in case the hardlink fails.

In case the destination already exists, it is forcibly removed first, to
avoid clobering any existing target file (and especially any hardlink to
it), since cp -f does not remove the destination file, but clobbers it.

In some situations, it will be necessary that the destination file is
named differently than the source, so if a third argument is specified,
it is treated as the basename of the destination file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-11 23:14:37 +02:00
Samuel Martin
bf8f0136e9 support/scripts/check-host-rpath: also check HOST_DIR/{bin, sbin}
At least syslinux is installing stuff in HOST_DIR/sbin.

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-04-21 22:17:36 +02:00
Samuel Martin
ac031cfb97 support/scripts: fix graph-depends when run with python3
Make graph-depends script opening the output file in text mode since
only ascii characters will be written.

This change fixes the following error occuring when the default host
python interpreter is python3:

  make: Entering directory '/opt/buildroot'
  Getting targets
  Getting dependencies for ['toolchain-external', 'toolchain', 'busybox', ...]
  Getting dependencies for ['host-python3', 'host-pkgconf', 'host-gettext', ...]
  Getting dependencies for ['host-libxml2', 'host-swig', 'host-m4', ...]
  Getting version for ['toolchain-external', 'toolchain', 'busybox', ...]
  Traceback (most recent call last):
    File "/opt/buildroot/support/scripts/graph-depends", line 425, in <module>
      outfile.write("digraph G {\n")
  TypeError: a bytes-like object is required, not 'str'
  Makefile:807: recipe for target 'graph-depends' failed
  make[1]: *** [graph-depends] Error 1
  Makefile:84: recipe for target '_all' failed
  make: *** [_all] Error 2
  make: Leaving directory '/opt/buildroot'

While with python2, adding 'b' to the openning mode has no effect on
Linux (c.f. [2]), the above error is expected with python3 (c.f. [1]).

Therefore, just open the outfile in default (i.e. text) mode.

[1] https://docs.python.org/3/library/functions.html#open
[2] https://docs.python.org/2/library/functions.html#open

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-04-18 11:34:33 +02:00
Francois Perrad
746a91ab31 scancpan: use recommend & test flags only at first level
Currently, these flags are recursively propagated. This behavior is
not expected by users, because it can cause dependencies explosively.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-03-15 23:16:33 +01:00
Francois Perrad
d6cff4abdc scancpan: handle recommended dependencies as optional packages
Currently, without the flag -recommend, scancpan takes as dependency
only one which has the relationship "requires"; this mode works fine.
And, with the flag -recommend, scancpan takes all ones (ie. with
relationship "requires" or "recommends") in the same way; this mode
never works fine, because it is too simplistic.

With this commit, the "not required" dependencies are handled as
optional BR package or skipped if a cyclic dependency is detected.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-03-15 23:16:29 +01:00
Thomas Petazzoni
ee8e4c5c25 graph-depends: add support for excluding host packages
Just like the --stop-on and --exclude options allow to stop on or
exclude virtual packages from the list by passing the "virtual" magic
value, this commit extends the graph-depends logic to support a "host"
magic value for --stop-on and --exclude. This will allow to draw the
graph by stopping on host packages, or by excluding host packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: minor code beautification suggested by Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-08 21:29:06 +01:00
Thomas Petazzoni
7771bb93b2 graph-depends: fix handling of "virtual" in exclude_list
The condition to determine if a virtual package should be excluded
from the list due to "virtual" being passed in --exclude is under a
loop iterating over each entry of the exclude_list, but it doesn't use
the iterator of this list.

Indeed, the condition contains:

	"virtual" in exclude_list

which checks automatically if "virtual" was passed in the list. Due to
this, there is no need for this check to be within the "for p in
exclude_list" iteration. This commit fixes that by moving the check
outside of the loop.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-08 21:28:19 +01:00
Yann E. MORIN
90551bfac0 support/graph-depends: teach it to only check dependencies
Add an option to graph-depends to only do the dependency checks and not
generate the dot program.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-07 23:21:40 +01:00
Yann E. MORIN
f48c08f0f5 support/graph-depends: detect circular dependencies
Currently, if there is a circular dependency in the packages, the
graph-depends script just errors out with a Python RuntimeError which is
not caught, resulting in a very-long backtrace which does not provide
any hint as what the real issue is (even if "RuntimeError: maximum
recursion depth exceeded" is a pretty good hint at it).

We fix that by recursing the dependency chain of each package, until we
either end up with a package with no dependency, or with a package
already seen along the current dependency chain.

We need to introduce a new function, check_circular_deps(), because we
can't re-use the existing ones:

  - remove_mandatory_deps() does not iterate,

  - remove_transitive_deps() does iterate, but we do not call it for the
    top-level package if it is not 'all'

  - it does not make sense to use those functions anyway, as they were
    not designed to _check_ but to _act_ on the dependency chain.

Since we've had time-related issues in the past, we do not want to
introduce yet another time-hog, so here are timings with the circular
dependency check:

    $ time python -m cProfile -s cumtime support/scripts/graph-depends
    [...]
             28352654 function calls (20323050 primitive calls) in 87.292 seconds

       Ordered by: cumulative time

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            1    0.012    0.012   87.292   87.292 graph-depends:24(<module>)
           21    0.000    0.000   73.685    3.509 subprocess.py:473(_eintr_retry_call)
            7    0.000    0.000   73.655   10.522 subprocess.py:768(communicate)
            7   73.653   10.522   73.653   10.522 {method 'read' of 'file' objects}
          5/1    0.027    0.005   43.488   43.488 graph-depends:164(get_all_depends)
            5    0.003    0.001   43.458    8.692 graph-depends:135(get_depends)
            1    0.001    0.001   25.712   25.712 graph-depends:98(get_version)
            1    0.001    0.001   13.457   13.457 graph-depends:337(remove_extra_deps)
         1717    1.672    0.001   13.050    0.008 graph-depends:290(remove_transitive_deps)
    9784086/2672326    5.079    0.000   11.363    0.000 graph-depends:274(is_dep)
    2883343/1980154    2.650    0.000    6.942    0.000 graph-depends:262(is_dep_uncached)
            1    0.000    0.000    4.529    4.529 graph-depends:121(get_targets)
      2883343    1.123    0.000    1.851    0.000 graph-depends:246(is_dep_cache_insert)
      9784086    1.783    0.000    1.783    0.000 graph-depends:255(is_dep_cache_lookup)
      2881580    0.728    0.000    0.728    0.000 {method 'update' of 'dict' objects}
            1    0.001    0.001    0.405    0.405 graph-depends:311(check_circular_deps)
    12264/1717    0.290    0.000    0.404    0.000 graph-depends:312(recurse)
    [...]
    real    1m27.371s
    user    1m15.075s
    sys     0m12.673s

The cumulative time spent in check_circular_deps is just below 0.5s,
which is largely less than 1% of the total run time.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-07 23:20:38 +01:00
Yann E. MORIN
a3f75bcc76 support/graph-depends: add option to specify output file
Currently, graph-depends outputs the dotfile program to stdout, and uses
stderr to trace the dependencies it is currently looking for.

Redirection was done because the output was directly piped into the dot
program to generate the final PDF/SVG/... dependency graph, but that
meant that an error in the graph-depends script was never caught
(because shell pipes only return the final command exit status, and an
empty dot program is perfectly valid so dot would not complain).

Add an option to tell graph-depends where to store the generated dot
program, and keep stdout as the default if not specified.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas: rename metavar from DOT_FILE to OUT_FILE for consistency with
the rest of the new option naming.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-07 23:15:30 +01:00
Thomas De Schampheleire
8c5afd120e support/scripts: add size-stats-compare script
Leverage the CSV files produces by size-stats (make graph-size) to allow
for a comparison of rootfs size between two different buildroot
compilations.

The script takes the file-size CSV files of two compilations as input, and
produces a textual report of the differences per package.
Using the -d/--detail flag, the report will show the file size changes
instead of package size changes.
The -t/--threshold option allows to ignore file size differences smaller
or equal than the given threshold (in bytes).

Example output is:

Size difference per package (bytes), threshold = 0
--------------------------------------------------------------------------------
       -8192         busybox
      228572   added dmalloc
      301584   added jq
--------------------------------------------------------------------------------
      521964         TOTAL

or with detailed view:

Size difference per file (bytes), threshold = 0
--------------------------------------------------------------------------------
       -8192         bin/busybox
       18152   added usr/bin/jq
       39252   added usr/bin/dmalloc
       46968   added usr/lib/libdmalloc.so
       47288   added usr/lib/libdmallocxx.so
       47316   added usr/lib/libdmallocth.so
       47748   added usr/lib/libdmallocthcxx.so
      283432   added usr/lib/libjq.so.1.0.4
--------------------------------------------------------------------------------
      521964         TOTAL

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-02-07 15:52:25 +01:00
Peter Korsgaard
53703ee67a support/scripts: drop ancient build-ext3-img script
It hasn't been updated since it was added in 2008, and nowadays things kind
of stuff should be handled with genimage.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-02-01 11:51:08 +01:00
Arnout Vandecappelle
c1a674f994 support/scripts/apply-patches.sh: fix whitespace
The apply-patches.sh script was using a mix of tabs and spaces, and
some three-space indentation. Normalize everything to four-space
indentation.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-01-13 22:09:08 +01:00
Thomas Petazzoni
a059055c52 support/scripts/size-stats: fix copy/paste error in ArgumentParser
A copy/paste error in the ArgumentParser() constructor call disclosed
the fact that the author of the script has shamefully based his work
on the existing graph-build-time script. This commit fixes this
mistake, therefore hiding in a better way how size-stats was
vampirized from graph-build-time.

Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-01-13 16:35:19 +01:00
Peter Korsgaard
f62ce058be graph-depends: correct is_dep() comment
The uncached variant is called is_dep_uncached(), not is_dep_full().

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-12-29 23:46:10 +01:00
Thomas Petazzoni
5ed60cee1e graph-depends: optimize remove_transitive_deps()
For large configurations, the execution time of
remove_transitive_deps() becomes really high, due to the number of
nested loops + the is_dep() function being recursive.

For an allyespackageconfig, the remove_extra_deps() function takes 334
seconds to execute, and the overall time to generate the .dot file is
6 minutes and 39 seconds. Here is a timing of the different
graph-depends steps and the overall execution time:

  Getting dependencies:   42.5735 seconds
  Turn deps into a dict:   0.0023 seconds
  Remove extra deps:     334.1542 seconds
  Get version:            22.4919 seconds
  Generate .dot:           0.0197 seconds

  real	6m39.289s
  user	6m16.644s
  sys	0m8.792s

By adding a very simple cache for the results of is_dep(), we bring
down the execution time of the "Remove extra deps" step from 334
seconds to just 4 seconds, reducing the overall execution time to 1
minutes and 10 seconds:

  Getting dependencies:  42.9546 seconds
  Turn deps into a dict:  0.0025 seconds
  Remove extra deps:      4.9643 seconds
  Get version:           22.1865 seconds
  Generate .dot:          0.0207 seconds

  real	1m10.201s
  user	0m47.716s
  sys	0m7.948s

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: Gustavo Zacarias <gustavo.zacarias@free-electrons.com>
[yann.morin.1998@free.fr:
  - rename is_dep() to is_dep_uncached(), keep existig code as-is
  - add is_dep() as a cached-version of is_dep_uncached()
  - use constructs more conform with 2to3
  - use exceptions (EAFP) rather than check-before-use (LBYL) to be more
    pythonist; that even decreases the duration yet a little bit more!
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2015-12-29 23:26:22 +01:00
Ryan Barnett
aab481c14e apply-patches: only use first field of line for series file
A series file for quilt has a valid syntax of:

  fixes/autoconf.diff -p1
  fixes/doc-html-local-css.diff -p1
  fixes/gnu-inline.diff -p1

However, with the current way that a series file is handled, it will
error out because the -p1 is tried as a file. This is because in the
for loop that iterates the files, we only look for comment lines. Then
each line is used within a bash for loop which uses spaces a
delimiter. In order to fix this, we should only use the string that
comes before a space in the series file.

Note that the format allows for any arbitrary depth to the -pN field.
But since we'll have only one package with -pN fields, and all will be
-p1, we for now always assume -p1. This will have to be fixed whenever
we get a package with other values.

Signed-off-by: Ryan Barnett <ryanbarnett3@gmail.com>
[yann.morin.1998@free.fr: expand comment about the format of a series
file and how we interpret it]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
CC: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-12 14:34:47 +01:00
Yann E. MORIN
f42e262199 support/check-host-rpath: remove trailing '/' in host dir
Despite the comment saying so, the trailing '/' in the host directory is
not removed. Note however that it is properly removed from extracted
RPATH tags.

This is not visible when the host directory is our default $(O)/host
location, but breaks for user-supplied external host directory, when
the user leaves a trailing slash in the path.

Fix that.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-12-02 23:28:14 +01:00
Yann E. MORIN
5c0c385226 core: check host executables have appropriate RPATH
When we build our host programs, and they depend on a host library we
also build, we want to ensure that program actually uses that library at
runtime, and not the one from the system.

We currently ensure that in two ways:
  - we add a RPATH tag that points to our host library directory,
  - we export LD_LIBRARY_PATH to point to that same directory.

With these two in place, we're pretty much confident that our host
libraries will be used by our host programs.

However, it turns our that not all the host programs we build end up
with an RPATH tag:
  - some packages do not use our $(HOST_LDFLAGS)
  - some packages' build system are oblivious to those LDFLAGS

In this case, there are two situations:
  - the program is not linked to one of our host libraries: it in fact
    does not need an RPATH tag [0]
  - the program actually uses one of our host libraries: in that case it
    should have had an RPATH tag pointing to the host directory.

For libraries, they only need an RPATH if they depend on another library
that is not installed in the standard library path. However, any system
library will already be in the standard library path, and any library we
install ourselves is in $(HOST_DIR)/usr/lib so already in RPATH.

We add a new support script that checks that all ELF executables have
a proper DT_RPATH (or DT_RUNPATH) tag when they link to our host
libraries, and reports those file that are missing an RPATH. If a file
missing an RPATH is an executable, the script aborts; if only libraries
are are missing an RPATH, the script does not abort.

[0] Except if it were to dlopen() it, of course, but the only program
I'm aware of that does that is openssl, and it has a correct RPATH tag.

[Peter: reworded as suggested by Arnout, fix HOT_DIR typo in comment]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-11-18 22:47:10 +01:00
Arnout Vandecappelle
7caf46439c apply-patches.sh: use everything from the series file
When a series file exists, we should use every file mentioned in it,
not just the ones ending with .patch or .diff. Also, there's no need
to uncompress anything if it's mentioned in a series file (the tools
that manipulate series files don't support compressed patches).

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Doug Kehn <rdkehn@yahoo.com>
Tested-by: Doug Kehn <rdkehn@yahoo.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-11-17 09:24:53 +01:00
Cédric Marie
474d39a1ff Makefile: Remove 'quiet' variable
'quiet' variable is set and exported, but it is not used. We can safely
remove it.

This variable is inherited from the Makefile of the Linux kernel, and
is not used in Buildroot.

In support/scripts/mkmakefile, 'quiet' value is checked, but the test
is always true ('quiet' is never set to silent_), so the test can be
removed as well.

Signed-off-by: Cédric Marie <cedric.marie@openmailbox.org>
Reviewed-by: "James Knight" <james.d.knight@live.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-29 00:04:05 +01:00
Francois Perrad
e2aff3d7f5 scancpan: README as default license file
add this heuristic when no specific license file is found

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-26 22:56:05 +01:00
Yann E. MORIN
f0c4456530 support/mkusers: allow comments in users tables
The format of the users table files is non trivial, so it is sometimes
handy to add comments explaining the syntax (or simply the reason for
the user) inline in the files.

Ignore empty lines and comment lines prefixed with '#' similar to shell
or makedevs files.

Packages that defined no user (the vast majority) would cause an empty
line to be present in the internal users table, hence the reason we
skipped empty usernames. Now that we ignore empty lines, we no longer
need to check for empty usernames.

Reported-by: Peter Korsgaard <jacmet@uclibc.org>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-10-18 17:52:21 +02:00
Thomas Petazzoni
598c80be8f support/scripts: add size-stats script
This new script uses the data collected by the step_pkg_size
instrumentation hook to generate a pie chart of the size contribution
of each package to the target root filesystem, and two CSV files with
statistics about the package size and file size. To achieve this, it
looks at each file in $(TARGET_DIR), and using the
packages-file-list.txt information collected by the step_pkg_size
hook, it determines to which package the file belongs. It is therefore
able to give the size installed by each package.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-10-17 16:12:59 +02:00
Francois Perrad
2e39ffcc6a pkg-perl: refactor perl infrastructure
the perl dependency of cpan module is no longer generated by scancpan,
but added at the infrastructure level

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-10-02 20:56:52 +01:00
Francois Perrad
ceb1214632 scancpan: fix detection of native module
When a module is native or depends of a native module, it must be
disabled for static builds via its Config.in

We detect native modules by looking at the filenames listed in the
MANIFEST. If there is a file which looks like it contains code that
much be compiled (e.g. .c, .h and so on...), then we exclude that
module (and its dependencies) from static builds.

That's what we tried to do so far, but failed when there was a
comment on the same line as the filename in the manifest, like so:
    foo-bar.c # Bla bla bla

Fix that by detecting either endof-line (as currently done) or
end-of-string.

For an example of failed build of perl-html-parser, see
http://autobuild.buildroot.net/results/128/128671dfa23d843698a63220c2fac1f44e1d5845/

[Thomas: use better commit log proposed by Yann E. Morin.]

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-09-20 15:14:10 +02:00
Francois Perrad
8498474ce3 scancpan: remove hack for Module-Build
with Perl 5.22, Module-Build is no longer a core module

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-09-06 22:48:49 +02:00
Francois Perrad
072986df1c perl: bump to version 5.22.0
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-09-06 22:48:42 +02:00
Yann E. MORIN
9d0d4c3cd2 docs/manual: fix generation of deprecated list
Since commit 5f117c3 (webkit: mark as deprecated), generation of the
manual has been broken.

This is because that commit added a deprecated dependency on a
prompt-less symbol, BR2_PACKAGE_WEBKIT_ARCH_SUPPORTS. However, the
generation script does not check that a symbol has a prompt before
it attempts to add it to the deprecated list. So, we end up with
traceback:

    Writing the virtual-packages list in:
            /home/ymorin/dev/buildroot/O/build/docs/manual/virtual-package-list.txt
    Traceback (most recent call last):
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 510, in <module>
        buildroot.print_list(list_name, dry_run=args.dry_run, output=output)
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 466, in print_list
        item_label=item_label)
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 126, in format_asciidoc_table
        enable_choice=enable_choice))
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 350, in _format_symbol_prompt_location
        return "| {0:<40} <| {1}\n".format(get_label_func(symbol),
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 458, in <lambda>
        get_label = lambda x: self._get_symbol_label(x, mark_depr)
      File "/home/ymorin/dev/buildroot/buildroot/support/scripts/gen-manual-lists.py", line 313, in _get_symbol_label
        label = symbol.get_prompts()[0]
    IndexError: list index out of range

However, we can not use the existing _is_deprecated filter function to
filter out symbols without prompts, because this function is also used
to add a '(deprecated)' tag in the man package list (not that it would
not work, but it does not seem /right/). Furthermore, it could also be
used (but is currently not) to build the list of virtual packages, which
do not have a prompt.

So, introduce a filter function, aptly named _is_deprecated_feature(),
to be used as the filter to find deprecated feature, and keep the
existing _is_deprecated() that can be used in any context to decide
whether a symbol is deprecated or not.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Acked-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-08-02 19:18:51 +02:00
Francois Perrad
6e9d322a11 scancpan: adjust package name
underscore is not allowed in BR package name.
this problem was found with the Perl module DB_File
which must give the BR package perl-db-file.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-07-23 22:36:18 +02:00
Yann E. MORIN
6bae1ac575 core/out-of-tree: fix Makefile wrapper
Commit 971faf8 (Makefile: fix out-of-tree builds with multiple targets
with 'all') renamed the default target to '_all' to avoid name-clashing.

In doing so, I forgot to also fix the instance in the .PHONY rule.

Fix that now.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-07-16 23:38:52 +02:00
Maxime Hadjinlian
f53d69614e graph-depends: Strip skeleton from dependency
skeleton being a mandatory dependency, we don't want all our packages to
have a link back to that node, the graph would be awful.

Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-07-14 18:10:58 +02:00
Guido Martínez
fd13247a22 scripts: mkmakefile: set umask before calling BR's makefile
Small optimization so we don't have another 'make' level (caused by the
umask fix) when running the generated makefile.

Signed-off-by: Guido Martínez <guido@vanguardiasur.com.ar>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-07-13 17:19:56 +02:00
Ulf Magnusson
954c09ae62 support: remove outdated note re. Python 3 support
Kconfiglib now runs as either Python 2 or Python 3.

Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-06-12 21:19:52 +02:00