This script, and its companion library, is more-or-less Buildroot's
equivalent to the kernel get_maintainer.pl script: it allows to get the
list of developers to whom a set of patches should be sent to.
To do so, it first relies on a text file, named DEVELOPERS, at the root
of the Buildroot source tree (added in a followup commit) to list the
developers and the files they are interested in. The DEVELOPERS file's
format is simple:
N: Firstname Lastname <email>
F: path/to/file
F: path/to/another/file
This allows to associate developers with the files they are looking
after, be they related to a package, a defconfig, a filesystem image, a
package infrastructure, the documentation, or anything else.
When a directory is given, the tool assumes that the developer handles
all files and subdirectories in this directory. For example
"package/qt5/" can be used for the developers looking after all the Qt5
packages.
Conventional shell patterns can be used, so "package/python-*" can be
used for the developers who want to look after all packages matching
"python-*".
A few files are recognized specially:
- .mk files are parsed, and if they contain $(eval
$(<something>-package)), the developer is assumed to be looking after
the corresponding package. This way, autobuilder failures for this
package can be reported directly to this developer.
- arch/Config.in.<arch> files are recognized as "the developer is
looking after the <arch> architecture". In this case, get-developer
parses the arch/Config.in.<arch> to get the list of possible BR2_ARCH
values. This way, autobuilder failures for this package can be
reported directly to this developer.
- pkg/pkg-<infra>.mk are recognized as "the developer is looking after
the <infra> package infrastructure. In this case, any patch that adds
or touches a .mk file that uses this infrastructure will be sent to
this developer.
Examples of usage:
$ ./support/scripts/get-developers 0001-ffmpeg-fix-bfin-build.patch
git send-email--to buildroot@buildroot.org --to "Luca Ceresoli <luca@lucaceresoli.net>" --to "Bernd Kuhls <bernd.kuhls@t-online.de>"
$ ./support/scripts/get-developers -p imx-lib
Arnout Vandecappelle <arnout@mind.be>
Gary Bisson <gary.bisson@boundarydevices.com>
$ ./support/scripts/get-developers -a bfin
Waldemar Brodkorb <wbx@openadk.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We currently have four lists of packages in the manual:
- the non-virtual target packages,
- the virtual target packages,
- the host packages,
- the deprecated features.
Those list take more than half of the manual. They do not serve much
purpose except to show off.
After the recent discussion on the list [0], remove them all.
We can now get rid of our biggish and complex generating script (and its
companion library kconfiglib).
[0] http://lists.busybox.net/pipermail/buildroot/2016-September/171199.html
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently all cmake packages depend on host-cmake. Unfortunately
host-cmake takes a long time to configure and build: almost 7 minutes
on a dual-core i5 with SSD. The time does not change even with ccache
enabled.
Indeed, building host-cmake is avoidable if it is already installed on
the build host: CMake is supposed to be quite portable, and the only
patch in Buildroot for the CMake package seems to only affect
target-cmake.
Thus we automatically skip building host-cmake and use the one on the
system if:
- cmake is available on the system and
- it is recent enough.
First, we leverage the existing infrastructure in
support/dependencies/dependencies.mk to find out whether there's a
suitable cmake executable on the system. Its path can be passed in the
BR2_CMAKE environment variable, otherwise it defaults to "cmake". If
it is enabled, found and suitable then we set BR2_CMAKE_HOST_DEPENDENCY
to empty; otherwise we set BR2_CMAKE_HOST_DEPENDENCY to 'host-cmake' and
override BR2_CMAKE with "$(HOST_DIR)/usr/bin/cmake" to revert to using
our own cmake (the old behaviour).
Then in pkg-cmake.mk we replace the hard-coded dependency on host-cmake
to using the BR2_CMAKE_HOST_DEPENDENCY variable, and we use $(BR2_CMAKE)
instead of $(HOST_DIR)/usr/bin/cmake.
Unlike what we do for host-tar and host-xzcat, for host-cmake we do
not add host-cmake to DEPENDENCIES_HOST_PREREQ. If we did, host-cmake
would be a dependency for _any_ package when it's not installed on the
host, even when no cmake package is selected.
Cmake versions older than 3.0 are affected by the bug described and
fixed in Buildroot in ef2c1970e4 ("cmake: add patch to fix Qt mkspecs
detection"). The bug was fixed in upstream CMake in version 3.0 [0].
Amongst all the cmake packages currently in Buildroot, the currently
highest version mentioned in cmake_minimum_required() is 3.1 (grantlee
and opencv3).
Thus we use 3.1 as the lowest required cmake for now, until a package is
bumped, or a new package added, with a higher required version.
[0] https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Davide Viti <zinosat@tiscali.it>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
[yann.morin.1998@free.fr:
- simplify logic in check-host-cmake.mk;
- set and use BR2_CMAKE_HOST_DEPENDENCY, drop USE_SYSTEM_CMAKE;
- bump to cmake 3.1 for grantlee and opencv;
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
pkg-utils.mk contains various definitions that are used in the package
infrastructures and packages themselves.
However, those definitions can be useful in other parts of Buildroot,
and are already used in a few places that are not related to the package
infrastructure. Also, $(sep) will be needed early in the Makefile when
we eventually support multiple br2-external trees.
Since this file only contains definitions, we can include it anytime.
So, consider that file to no longer be specific to the package infras:
- move it to support and rename it,
- move a few similar definitions from the main Makefile to that file.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This adds support to pass options to the underlying command that is used
by downloader. Useful for retrieving data with server-side checking for
user login or passwords, use a proxy or use specific options for cloning
a repository via git and hg.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This change enforces the CMAKE_SYSROOT value set in the toolchainfile.cmake.
This fix overrides the CMake heuristics used to guess it, and turns off some
non-desirable behavior adding "-isystem ..." flags to the compiler command
line, misleading the compiler and making the build failed due to some
unfound standard headers.
Fixes:
http://autobuild.buildroot.net/results/f7e/f7e92678e91a6cb15ccf32d4a7d75b39f49d6000/defconfig
(and others)
Cc: Ben Boeckel <mathstuf@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The size-stats script fails when the usb_modeswitch_data is enabled,
because this package installs files that contain commas in their
name. However, the size-stats script also uses comma as a separator for
its CSV files, causing a "ValueError: too many values to unpack" in:
pkg, fpath = l.split(",")
Fix this by splitting only the two fields that need to be split.
The bug was reported by Matthias <porto.rio@gmx.net>, who also suggested
a fix.
Fixes bug #9136.
Reported-by: Matthias <porto.rio@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The "--no-patch" option used by the git downloader appeared on git
1.8.4. Systems with older git versions show an error and fall back to
the wget downloader, which isn't suitable for all the cases.
Signed-off-by: Enrique Ocaña González <eocanha@igalia.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since efl update to 1.15 version, the efl package is a "real"
Buildroot package. It doesn't contain any subdirectories anymore.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
As reported by Sébastien Szymanski [1], the apply-patches script
doesn't stop if a tar command can't extract an archive.
Use "set -e" to exit immediately if a command return an error.
Be sure to ignore any expected error: when we check if a patch to be
applied has the same basename as an already applied patch, the grep
would fail when no such patch was already applied. We should not fail
in this case.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
As reported by Sébastien Szymanski [1], the apply-patches script
doesn't stop if a tar command can't extract an archive.
Use "set -e" to exit immediately if a command return an error.
[1] http://patchwork.ozlabs.org/patch/626196
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
[Thomas:
- add comment in scancpan about the version dependency, suggested by
Yann E. Morin.
- add comment in perl.mk about the need to sync any version change with
scancpan, also suggested by Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
sha256 checksum will be computed locally either by scanpypi at package
creation or by hand by package updates. Define this checksum as
'computed locally' so that one doesn't need to change this comment by
package updates. Also put comments for both md5 and sha256 in one line.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Make sure a help text is terminated with a full stop.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Since the fortran support is conditional, only enable it when needed.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Cc: Benjamin Kamath <bkamath@spaceflight.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add a new package variable that packages can set to specify that they
need git submodules.
Only accept this option if the download method is git, as we can not get
submodules via an http download (via wget).
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Aleksandar Simeonov <aleksandar@barix.com>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Tested-By: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some git repositories may be split into a master repository and
submodules. Up until now, we did not have support for submodules,
because we were using bare clones, in which it is not possible to
update the list of submodules.
Now that we are using plain clones with a working copy, we can retrieve
the submdoules.
Add an option to the git download helper to kick the update of
submodules, so that they are only fetched for those packages that
require them. Also document the existing -q option at the same time.
Submodules have a .git file at their root, which contains the path to
the real .git directory of the master repository. Since we remove it,
there is no point in keeping those .git files either.
Note: this is currently unused, but will be enabled with the follow-up
patch that adds the necessary parts in the pkg-generic and pkg-download
infrastructures.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would
have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the
generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, we are using bare clones, so as to minimise the disk usage,
most notably for largeish repositories such as the one for the Linux
kernel, which can go beyond the 1GiB barrier.
However, this precludes updating (and thus using) the submodules, if
any, of the repositories, as a working copy is required to use
submodules (becaue we need to know the list of submodules, where to find
them, where to clone them, what cset to checkout, and all those is
dependent upon the checked out cset of the father repository).
Switch to using /plain/ clones with a working copy.
This means that the extra refs used by some forges (like pull-requests
for Github, or changes for gerrit...) are no longer fetched as part of
the clone, because git does not offer to do a mirror clone when there is
a working copy.
Instead, we have to fetch those special refs by hand. Since there is no
easy solution to know whether the cset the user asked for is such a
special ref or not, we just try to always fetch the cset requested by
the user; if this fails, we assume that this is not a special ref (most
probably, it is a sha1) and we defer the check to the archive creation,
which would fail if the requested cset is missing anyway.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In most cases Python's package dependencies found in setup.py are
runtime dependencies and hence don't need to be mentioned in *.mk
file.
Also add '# runtime' tag to select statements in Config.in.
__create_mk_requirements() itself is left for future uses (cffi backend
handling etc.).
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, the legal-info infra only saves the source archive of a
package. However, that's not enough as we may apply some patches on
packages sources.
We do suggest users to also redistribute the Buildroot sources as part
of their compliance distribution, so the patches bundled in Buildroot
would indeed be included in the compliance distribution.
However, that's still not enough, since we may download some patches, or
the user may use a global patch directory. Patches in there might not
end up in the compliance distribution, and there are risks of
non-conformity.
So, always include patches alongside the source archive.
To ensure reproducibility, we also generate a series file, so patches
can be re-applied in the correct order.
We get the list of patches to include from the list of patches that were
applied by the package infrastructure (via the apply-patches support
script). So, we need to get packages properly extracted and patched
before we can save their legal-info, not just in the case they define
_LICENSE_FILES.
Update the legal-info header accordingly.
Note: this means that, when a package is not patched and defines no
LICENSE_FILES, we will extract and patch it for nothing. There is no
easy way to know whether we have to patch a package or not. We can only
either duplicate the logic to detect patches (bad) or rely on the infra
actually patching the package. Also, a vast majority of packages are
either patched, or define _LICENSE_FILES, so it is best and easiest to
always extract and patch them prior to legal-info.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Patches we save can come from various locations:
- bundled with Buildroot
- downloaded
- from one or more global-patch-dir
It is possible that two patches lying into different locations have the
same basename, like so (first is bundled, second is from an hypothetical
global-patch-dir):
package/foo/0001-fix-Makefile.patch
/path/to/my/patches/foo/0001-fix-Makefile.patch
In that case, when running legal-info, we'd save only the second patch,
overwriting the first. That would be problematic, because:
- either the second patch depends on the first, and thus would no longer
apply (this is easy to detect, though),
- or the second patch does not depend on the first, and the compliance
delivery will not be complete (this is much harder to detect).
We fix that by checking that no two patches have the same same basename.
If we find that the basename of the patch to be applied collides with
that of a previously applied patch, we error out and report the duplicate.
The unfortunate side-effect is that existing setups will now break in
that situation, but that's a minor, corner-case issue that is easily
fixed.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: adjust coding style, fix minor typos in the commit log.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we only store the filename of the applied patches.
However, we are soon to want to install those patches in the legal-info
directory, so we'll have to know where those patches come from.
Instead of duplicating the logic to find the patches (bundled,
downloaded, from a global patch dir...), just store the full path to
each of those patches so we can retrieve them more easily later on.
Also always create the list-file, even if empty, so that we need not
test for its existence before reading it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
[Tested only with patches in the Buildroot sources]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: used $PWD instead of $(pwd), as suggested by Arnout.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
A utility for creating python package from the python package index.
It fetches packages info from http://pypi.python.org and generates
corresponding packages files.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
Tested-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
[Thomas: minor tweaks.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, each python package (be it the python interpreter package
itself or external python modules) is responsible for compiling its
.py into .pyc files. Unfortunately, this is not ideal as some packages
only install .py files without compiling them into .pyc files. In this
case, if the Buildroot configuration specifies to keep only the .pyc
files, the .py files are removed and lost.
To address this, this commit changes the logic by making the
compilation of .pyc files a global operation: the python interpreter
packages register a target finalize hook that is in charge of
compiling all installed .py files.
The *.pyc generation on a per package basis is disabled in the
python-package infrastructure by passing the "--no-compile" option to
setup.py.
The *.pyc generation for the Python interpreter internal modules is
disabled through --disable-pyc-build configure option.
A small helper script is used to perform the compilation, the purpose
of this script is to abort the compilation process if one of the .py
file cannot be compiled. It has been provided by Samuel Martin and
integrated into this commit.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas:
- rework for python 3.5
- integrate Samuel proposal that allows to detect compilation
failures.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The graph-build-time help text currently looks like this:
usage: graph-build-time [-h] [--type GRAPH_TYPE] [--order GRAPH_ORDER]
[--alternate-colors] [--input OUTPUT] --output OUTPUT
Obviously, naming the parameter for --input as OUTPUT is not a very
good idea, so this commit fixes that to name it "INPUT", as expected.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When preparing the legal-info, the source archives are copied in the
legal-info/ output directory. When the archives are big, it can take
quite a bit of time and unnecessarily uses disk space. When the
legal-info output directory is on the same filesystem as the BR2_DL_DIR,
we can easily reduce copy time and disk usage by just using hardlins
instead of copying. However, the BR2_DL_DIR may be on a different
filesystem, so we must fallback to copying in this case
Introduce a helper script that copies a source file into a destination
directory, by first attempting to hard-link, and falling back to a
plain copy in case the hardlink fails.
In case the destination already exists, it is forcibly removed first, to
avoid clobering any existing target file (and especially any hardlink to
it), since cp -f does not remove the destination file, but clobbers it.
In some situations, it will be necessary that the destination file is
named differently than the source, so if a third argument is specified,
it is treated as the basename of the destination file.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
At least syslinux is installing stuff in HOST_DIR/sbin.
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Make graph-depends script opening the output file in text mode since
only ascii characters will be written.
This change fixes the following error occuring when the default host
python interpreter is python3:
make: Entering directory '/opt/buildroot'
Getting targets
Getting dependencies for ['toolchain-external', 'toolchain', 'busybox', ...]
Getting dependencies for ['host-python3', 'host-pkgconf', 'host-gettext', ...]
Getting dependencies for ['host-libxml2', 'host-swig', 'host-m4', ...]
Getting version for ['toolchain-external', 'toolchain', 'busybox', ...]
Traceback (most recent call last):
File "/opt/buildroot/support/scripts/graph-depends", line 425, in <module>
outfile.write("digraph G {\n")
TypeError: a bytes-like object is required, not 'str'
Makefile:807: recipe for target 'graph-depends' failed
make[1]: *** [graph-depends] Error 1
Makefile:84: recipe for target '_all' failed
make: *** [_all] Error 2
make: Leaving directory '/opt/buildroot'
While with python2, adding 'b' to the openning mode has no effect on
Linux (c.f. [2]), the above error is expected with python3 (c.f. [1]).
Therefore, just open the outfile in default (i.e. text) mode.
[1] https://docs.python.org/3/library/functions.html#open
[2] https://docs.python.org/2/library/functions.html#open
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, these flags are recursively propagated. This behavior is
not expected by users, because it can cause dependencies explosively.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, without the flag -recommend, scancpan takes as dependency
only one which has the relationship "requires"; this mode works fine.
And, with the flag -recommend, scancpan takes all ones (ie. with
relationship "requires" or "recommends") in the same way; this mode
never works fine, because it is too simplistic.
With this commit, the "not required" dependencies are handled as
optional BR package or skipped if a cyclic dependency is detected.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The way we use it, gzip will store the current time in the header, which
leads to unreproducible archives.
Fix that by telling gzip to not store the name and date of the file it
compresses, with the -n option. Since it compresses its stdin, there was
already no filename stored; now there's even no date stored.
Note: gzip has had -n since at least 1.2.4, released in 1993, so
virtually every gzip out there nowadays has it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Just like the --stop-on and --exclude options allow to stop on or
exclude virtual packages from the list by passing the "virtual" magic
value, this commit extends the graph-depends logic to support a "host"
magic value for --stop-on and --exclude. This will allow to draw the
graph by stopping on host packages, or by excluding host packages.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: minor code beautification suggested by Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The condition to determine if a virtual package should be excluded
from the list due to "virtual" being passed in --exclude is under a
loop iterating over each entry of the exclude_list, but it doesn't use
the iterator of this list.
Indeed, the condition contains:
"virtual" in exclude_list
which checks automatically if "virtual" was passed in the list. Due to
this, there is no need for this check to be within the "for p in
exclude_list" iteration. This commit fixes that by moving the check
outside of the loop.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add an option to graph-depends to only do the dependency checks and not
generate the dot program.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, if there is a circular dependency in the packages, the
graph-depends script just errors out with a Python RuntimeError which is
not caught, resulting in a very-long backtrace which does not provide
any hint as what the real issue is (even if "RuntimeError: maximum
recursion depth exceeded" is a pretty good hint at it).
We fix that by recursing the dependency chain of each package, until we
either end up with a package with no dependency, or with a package
already seen along the current dependency chain.
We need to introduce a new function, check_circular_deps(), because we
can't re-use the existing ones:
- remove_mandatory_deps() does not iterate,
- remove_transitive_deps() does iterate, but we do not call it for the
top-level package if it is not 'all'
- it does not make sense to use those functions anyway, as they were
not designed to _check_ but to _act_ on the dependency chain.
Since we've had time-related issues in the past, we do not want to
introduce yet another time-hog, so here are timings with the circular
dependency check:
$ time python -m cProfile -s cumtime support/scripts/graph-depends
[...]
28352654 function calls (20323050 primitive calls) in 87.292 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.012 0.012 87.292 87.292 graph-depends:24(<module>)
21 0.000 0.000 73.685 3.509 subprocess.py:473(_eintr_retry_call)
7 0.000 0.000 73.655 10.522 subprocess.py:768(communicate)
7 73.653 10.522 73.653 10.522 {method 'read' of 'file' objects}
5/1 0.027 0.005 43.488 43.488 graph-depends:164(get_all_depends)
5 0.003 0.001 43.458 8.692 graph-depends:135(get_depends)
1 0.001 0.001 25.712 25.712 graph-depends:98(get_version)
1 0.001 0.001 13.457 13.457 graph-depends:337(remove_extra_deps)
1717 1.672 0.001 13.050 0.008 graph-depends:290(remove_transitive_deps)
9784086/2672326 5.079 0.000 11.363 0.000 graph-depends:274(is_dep)
2883343/1980154 2.650 0.000 6.942 0.000 graph-depends:262(is_dep_uncached)
1 0.000 0.000 4.529 4.529 graph-depends:121(get_targets)
2883343 1.123 0.000 1.851 0.000 graph-depends:246(is_dep_cache_insert)
9784086 1.783 0.000 1.783 0.000 graph-depends:255(is_dep_cache_lookup)
2881580 0.728 0.000 0.728 0.000 {method 'update' of 'dict' objects}
1 0.001 0.001 0.405 0.405 graph-depends:311(check_circular_deps)
12264/1717 0.290 0.000 0.404 0.000 graph-depends:312(recurse)
[...]
real 1m27.371s
user 1m15.075s
sys 0m12.673s
The cumulative time spent in check_circular_deps is just below 0.5s,
which is largely less than 1% of the total run time.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, graph-depends outputs the dotfile program to stdout, and uses
stderr to trace the dependencies it is currently looking for.
Redirection was done because the output was directly piped into the dot
program to generate the final PDF/SVG/... dependency graph, but that
meant that an error in the graph-depends script was never caught
(because shell pipes only return the final command exit status, and an
empty dot program is perfectly valid so dot would not complain).
Add an option to tell graph-depends where to store the generated dot
program, and keep stdout as the default if not specified.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas: rename metavar from DOT_FILE to OUT_FILE for consistency with
the rest of the new option naming.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Leverage the CSV files produces by size-stats (make graph-size) to allow
for a comparison of rootfs size between two different buildroot
compilations.
The script takes the file-size CSV files of two compilations as input, and
produces a textual report of the differences per package.
Using the -d/--detail flag, the report will show the file size changes
instead of package size changes.
The -t/--threshold option allows to ignore file size differences smaller
or equal than the given threshold (in bytes).
Example output is:
Size difference per package (bytes), threshold = 0
--------------------------------------------------------------------------------
-8192 busybox
228572 added dmalloc
301584 added jq
--------------------------------------------------------------------------------
521964 TOTAL
or with detailed view:
Size difference per file (bytes), threshold = 0
--------------------------------------------------------------------------------
-8192 bin/busybox
18152 added usr/bin/jq
39252 added usr/bin/dmalloc
46968 added usr/lib/libdmalloc.so
47288 added usr/lib/libdmallocxx.so
47316 added usr/lib/libdmallocth.so
47748 added usr/lib/libdmallocthcxx.so
283432 added usr/lib/libjq.so.1.0.4
--------------------------------------------------------------------------------
521964 TOTAL
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
