Just like the --stop-on and --exclude options allow to stop on or
exclude virtual packages from the list by passing the "virtual" magic
value, this commit extends the graph-depends logic to support a "host"
magic value for --stop-on and --exclude. This will allow to draw the
graph by stopping on host packages, or by excluding host packages.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: minor code beautification suggested by Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The condition to determine if a virtual package should be excluded
from the list due to "virtual" being passed in --exclude is under a
loop iterating over each entry of the exclude_list, but it doesn't use
the iterator of this list.
Indeed, the condition contains:
"virtual" in exclude_list
which checks automatically if "virtual" was passed in the list. Due to
this, there is no need for this check to be within the "for p in
exclude_list" iteration. This commit fixes that by moving the check
outside of the loop.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add an option to graph-depends to only do the dependency checks and not
generate the dot program.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, if there is a circular dependency in the packages, the
graph-depends script just errors out with a Python RuntimeError which is
not caught, resulting in a very-long backtrace which does not provide
any hint as what the real issue is (even if "RuntimeError: maximum
recursion depth exceeded" is a pretty good hint at it).
We fix that by recursing the dependency chain of each package, until we
either end up with a package with no dependency, or with a package
already seen along the current dependency chain.
We need to introduce a new function, check_circular_deps(), because we
can't re-use the existing ones:
- remove_mandatory_deps() does not iterate,
- remove_transitive_deps() does iterate, but we do not call it for the
top-level package if it is not 'all'
- it does not make sense to use those functions anyway, as they were
not designed to _check_ but to _act_ on the dependency chain.
Since we've had time-related issues in the past, we do not want to
introduce yet another time-hog, so here are timings with the circular
dependency check:
$ time python -m cProfile -s cumtime support/scripts/graph-depends
[...]
28352654 function calls (20323050 primitive calls) in 87.292 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.012 0.012 87.292 87.292 graph-depends:24(<module>)
21 0.000 0.000 73.685 3.509 subprocess.py:473(_eintr_retry_call)
7 0.000 0.000 73.655 10.522 subprocess.py:768(communicate)
7 73.653 10.522 73.653 10.522 {method 'read' of 'file' objects}
5/1 0.027 0.005 43.488 43.488 graph-depends:164(get_all_depends)
5 0.003 0.001 43.458 8.692 graph-depends:135(get_depends)
1 0.001 0.001 25.712 25.712 graph-depends:98(get_version)
1 0.001 0.001 13.457 13.457 graph-depends:337(remove_extra_deps)
1717 1.672 0.001 13.050 0.008 graph-depends:290(remove_transitive_deps)
9784086/2672326 5.079 0.000 11.363 0.000 graph-depends:274(is_dep)
2883343/1980154 2.650 0.000 6.942 0.000 graph-depends:262(is_dep_uncached)
1 0.000 0.000 4.529 4.529 graph-depends:121(get_targets)
2883343 1.123 0.000 1.851 0.000 graph-depends:246(is_dep_cache_insert)
9784086 1.783 0.000 1.783 0.000 graph-depends:255(is_dep_cache_lookup)
2881580 0.728 0.000 0.728 0.000 {method 'update' of 'dict' objects}
1 0.001 0.001 0.405 0.405 graph-depends:311(check_circular_deps)
12264/1717 0.290 0.000 0.404 0.000 graph-depends:312(recurse)
[...]
real 1m27.371s
user 1m15.075s
sys 0m12.673s
The cumulative time spent in check_circular_deps is just below 0.5s,
which is largely less than 1% of the total run time.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, graph-depends outputs the dotfile program to stdout, and uses
stderr to trace the dependencies it is currently looking for.
Redirection was done because the output was directly piped into the dot
program to generate the final PDF/SVG/... dependency graph, but that
meant that an error in the graph-depends script was never caught
(because shell pipes only return the final command exit status, and an
empty dot program is perfectly valid so dot would not complain).
Add an option to tell graph-depends where to store the generated dot
program, and keep stdout as the default if not specified.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas: rename metavar from DOT_FILE to OUT_FILE for consistency with
the rest of the new option naming.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
For large configurations, the execution time of
remove_transitive_deps() becomes really high, due to the number of
nested loops + the is_dep() function being recursive.
For an allyespackageconfig, the remove_extra_deps() function takes 334
seconds to execute, and the overall time to generate the .dot file is
6 minutes and 39 seconds. Here is a timing of the different
graph-depends steps and the overall execution time:
Getting dependencies: 42.5735 seconds
Turn deps into a dict: 0.0023 seconds
Remove extra deps: 334.1542 seconds
Get version: 22.4919 seconds
Generate .dot: 0.0197 seconds
real 6m39.289s
user 6m16.644s
sys 0m8.792s
By adding a very simple cache for the results of is_dep(), we bring
down the execution time of the "Remove extra deps" step from 334
seconds to just 4 seconds, reducing the overall execution time to 1
minutes and 10 seconds:
Getting dependencies: 42.9546 seconds
Turn deps into a dict: 0.0025 seconds
Remove extra deps: 4.9643 seconds
Get version: 22.1865 seconds
Generate .dot: 0.0207 seconds
real 1m10.201s
user 0m47.716s
sys 0m7.948s
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: Gustavo Zacarias <gustavo.zacarias@free-electrons.com>
[yann.morin.1998@free.fr:
- rename is_dep() to is_dep_uncached(), keep existig code as-is
- add is_dep() as a cached-version of is_dep_uncached()
- use constructs more conform with 2to3
- use exceptions (EAFP) rather than check-before-use (LBYL) to be more
pythonist; that even decreases the duration yet a little bit more!
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
skeleton being a mandatory dependency, we don't want all our packages to
have a link back to that node, the graph would be awful.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
For clarity, this commit renames the TARGETS variable to the more
meaningful PACKAGES variable. Indeed, only packages (handled by one of
the package infrastructures) should be listed in this variable, and
not other random non-package targets.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Like for --stop-on, make --exclude recognise the keyword 'virtual',
to stop on virtual packages (as explained in the help...).
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Similar to --stop-on, but also omits the package from the graph.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Francois Perrad <fperrad@gmail.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Francois Perrad <fperrad@gmail.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add a new option to graph-depends, that users can set to stop the graph
on a specific (set of) package(s).
This accepts any actual package name, or the 'virtual' keyword to stop
on virtual packages.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Francois Perrad <fperrad@gmail.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Most of targets listed in TARGET_EXCEPTIONS these days are long
gone, so why still keep them?
Most of those targets were removed in this commit:
http://git.buildroot.net/buildroot/commit/?id=02b88600312554bf166f6cfd71f7f2ede783096a
Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@sunsite.dk>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
virtual packages are found by their version,
so we retrieve the version of all packages
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
It hides any error messages reported by make.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: Samuel Martin <s.martin49@gmail.com>
Tested-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The graph-depends script calls make. If the outer make was called
recursively, or if it was called with '-C <somedir>', then the
environment will contain "MAKEFLAGS=w --". Therefore, the recursive
make prints 'Entering' and 'Leaving' messages, which clobbers the
output for dot.
To avoid this, add "--no-print-directory" to the recursive make
arguments. Since we require GNU make 3.81, we can be sure that this
option is available.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This patch is the result of 2to3.
In addition, universal_newlines=True is added to the Popen calls. In
python3, this makes sure that the output is decoded so that we get a
string instead of a buffer object.
Cc: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The transitive dependencies make the graphs barely readable for large
configs, with a large number of packages.
So, just switch to not drawing the transitive dependencies by default.
By popular demand... ;-)
[Peter: reword]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc; Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This is ugly, since Python does not have enum constructs, so by moving
the 'type' of the constant ('MODE' here) to the beginning, we get an
artificial 'namespace' for the constants.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Although unnecessary (we already have initialisation via the parser),
initialise the 'transitive' option, and document it at the same time.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add some comment as well, enhance help text.
[thanks to Samuel for the hints to make it even more pythonic]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Merge the redundant-dependencies elimination into the newly introduced
transitive-dependencies elimination.
This makes the code cleaner and much shorter, because:
- the ('all',pkg) redundant dependency is in fact a transitive
dependency, and we now have code to deal with that
- the (pkg,'toolchain') dependency is easy enough to deal with that
having a separate function for that is overkill
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, all the dependencies of a package are drawn on the dependency
graph, including transitive dependencies (e.g. A->B->C and A->C).
For very big graphs, with lots of packages with lots of dependencies, the
dependency graph can be very dense, and transitive dependencies are
cluttering the graph.
In some cases, only getting the "build-order" dependencies is enough (e.g.
to see what impact a package rebuild would have).
Add a new environment variable to disable drawing transitive dependencies.
Basically, it would turn this graph:
pkg1 ---> pkg2 ---> pkg3 -------------------.
|\__________/ \ \
|\____________________ \ \
| \ \ \
`-> pkg4 ---> pkg5 ---> pkg6 ---> pkg7 ---> pkg8
\__________/
into that graph:
pkg1 ---> pkg2 ---> pkg3 -----------.
| \
`-> pkg4 ---> pkg5 ---> pkg6 ---> pkg7 ---> pkg8
[Thanks to Samuel for the parser hints]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Do not use the same colors for toolchain, host and target packages.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
[yann.morin.1998@free.fr rephrase commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
As Samuel said:
In Python, None is a singleton, and it is recommended to use "is" or
"is not" for testing them [1].
[1] http://legacy.python.org/dev/peps/pep-0008/#programming-recommendations
Reported-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Al packages depend on 'toolchain'. Currently, 'graph-depends' graphs this
dependency. The resulting graph is thus cluttered with less-than-useful
information.
Instead, do not graph the 'toolchain' dependency for any package, save
for the fake 'all' package. The graph is now a bit more readable.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, the complete dependency chain of a package is used to
generate the dependency graph. When this dependency chain is long,
the generated graph becomes almost unreadable.
However, it is often sufficient to get the first few levels of
dependency of a package.
Add a new variable BR2_GRAPH_DEPTH, that the user can set to limit
the depth of the dependency list.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we are using a crude, ad-hoc parsing of argv[].
This is a limiting factor to adding new options.
Use argparse instead, and introduce a single argument for now:
--package, -p PACKAGE
In the (near) future, we'll be able to add more option arguments,
such as depth-limiting for big graphs.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
We ran into a "Login incorrect" problem when running the same rootfs
image across platforms with different loging ports ttyS0/1/2/3.
Simply assignning "console" to BR2_TARGET_GENERIC_GETTY_PORT, which in
turn modifies the /etc/inittab, is not enough because the "console" device
was missing in the /etc/securetty.
While current securetty has enumerated a lot of ttys, this patch should save
some efforts to enumerate more.
[Peter: guard with single quotes]
Signed-off-by: Tzu-Jung Lee <tjlee@ambarella.com>
Signed-off-by: Spenser Gilliland <spenser@gillilanding.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
Otherwise, graph-depends tries to call 'make target-purgelocales-show-depends',
which does not exist, as 'target-purgelocales' is not an actual package.
Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
The graph-depends script tries to call 'make target-generic-dont-remount-rw',
which doesn't exist since 'target-generic-dont-remount-rw' is not a package.
See also the comments for commit 72bd61e5b8c2094378.
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
Since the introduction of the post-image mechanism, the graph-depends
script is broken: it tries to call 'make
target-post-image-show-depends', which doesn't exist since
'target-post-image' is not a package.
So we should simply ignore this 'target-post-image'.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
Add the root-password internal target to the exclusion list.
Fixes failures like:
Getting dependencies for [... 'target-root-passwd' ...]
Error getting dependencies [... 'target-root-passwd' ...]
Which is easily singled out with:
$ make target-root-passwd-show-depends
make[1]: *** No rule to make target `target-root-passwd-show-depends'.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
The "unknown" packages mechanism was used to render packages that did
not implement the make <pkg>-show-depends target, i.e the packages
that were not yet converted to one of the package infrastructures.
Since now all packages have been converted, we can remove this
"unknown" packages feature.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
Since 9bc7b1d4ae, all X.org .mk files
are parsed unconditionally, even if BR2_PACKAGE_XORG7 is disabled.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
Until now, graph-depends was calling "make <pkg>-show-depends"
individually for eack package, which was very slow. Now, it calls
"make <pkg1>-show-depends <pkg2>-show-depends ... <pkgN>-show-depends"
for all packages it knows, and then does that recursively. It reduces
the number of make invocations to the deepest dependency chain in the
current configuration, instead of having a number of make invocations
equal to the number of enabled packages.
For a configuration with xvkbd enabled (which brings a significant
number of X.org dependencies) and a tar root filesystem, the time to
execute graph-depends was:
real 5m14.944s
user 4m53.590s
sys 0m14.069s
After our optimizations, it is now:
real 0m33.096s
user 0m30.878s
sys 0m1.472s
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
In preparation for more graph-depends improvements, use a
TARGET_EXCEPTIONS list to list all the targets that should be ignored
while building the dependency graph.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
When doing a full graph of the dependencies, graph-depends starts by
doing a "make show-targets", which lists all the packages registered
in the $(TARGETS) variable. This variable contains all packages that
are enabled according to the .config file. Then, for each of those
packages, we used to create a "all" -> "package" dependency, even if
in fact most of some packages are already dependencies of other
packages. This creates a needlessly complex dependency graph.
This patch modifies graph-depends so that it filters out the unneeded
"all" -> "package" dependencies when "package" is already the
dependency of another package.
For example, if you have a configuration with libpng (which selects
zlib), "make show-targets" displays "libpng zlib", so graph-depends
used to create the following dependencies: (all -> libpng, all ->
zlib, libpng -> zlib). However, the (all -> zlib) dependency is not
really needed, as zlib is already the dependency of libpng. Those
dependencies are now filtered out.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>
graph-depends calls make to get the list of packages, and the
dependencies of each package.
When called out-of-tree, the Makefile is a wrapper that calls
the real Makefile, so make will spit out a line like:
make -C /path/to/buildroot O=/path/to/build-dir show-targets
which graph-depends wrongly believes is part of the target list.
Be silent when calling make, as we really only want the target
and dependency lists.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
