Evaluating all the <PKG>_RECURSIVE_FINAL_DEPENDENCIES variables
(abbreviated RFD hereafter) ends up being quite slow. Enough, on a
reasonable modern workstation, to increase the time it takes to run
"make printvars" from 13 seconds in 2018.02 to 371 seconds in 2019.02.
This patch improves this by using dynamic programming to speed the
evaluation of RFD, reducing the before mentioned printvars time to about
14.6 seconds.
The evaluation of PKG1_RFD requires recursively evaluating each of
PKG1's dependencies' RFDs, then their dependencies' RFDs, and so on.
The same is done for PKG2_RFD. But it's likely that many of the
dependencies of PKG2 are the same as PKG1. And when we consider all
packages, the dependencies are re-computed many thousands of times.
To avoid this re-computation we memoize, or save, the computed value of
each RFD variable when it found the first time. Subsequent evaluations
re-use the memoized value.
Surprisingly, this ends up being not all the hard to implement in make.
The basic construct is this:
VAR = $(if !defined(VAR__X),$(eval VAR__X := value))$(VAR__X)
The first time VAR is evaluated VAR__X will not be defined, and code to
set VAR__X to the computed value is eval'd. Then the now defined value
of VAR__X is returned. Subsequent evaluations can just return VAR__X.
It is important to note that VAR is defined with '=', as not enough
information (namely, all packages' dependencies) is know when it is
parsed to find the correct value. VAR will be evaluated each time it is
used. But VAR__X is defined with ":=", so that it is evaluated once
when defined, and not each time it is used.
Signed-off-by: Trent Piepho <tpiepho@impinj.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In commit 7fb6e78254 (core/instrumentation: shave minutes off the
build time), the built stampfile is used as a reference to detect files
installed by a package.
However, packages may install files keeping their mtime intact, and we
end up not detecting this. For example, the internal skeleton package
will install (e.g.) /etc/passwd with an mtime of when the file was
created in $(TOP_DIR), which could be the time the git repository was
checked out; that mtime is always older than the build stamp file, so
files installed by the skeleton package are never accounted for to that
package, or to any other package for that matters.
We switch to an alternate solution, which consists of storing some extra
metadata per file, so that we can more reasily detect modifications to
the files. Then we compare the state before the package is installed (by
reusing the existing list) and after the package is installed, compare
that to list any new file or modified files (in reality, ignoring
untouched and removed files). Finally, we store the file->package
association in the global list and store the new stat list as the global
list.
The format used for the .stat file is:
mtime:inode:perms:filetype:size,filename
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
[Peter: rename files, reformat, only look for files and symlinks and pass
LC_ALL=C to comm as pointed out by Thomas De Schampheleire]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
realpath is missing on oldish distributions, like Debian 7, which is
still used in the wild.
Use readlink instead; that has been available since the dawn of ages now
(well, coreutils had it in 2003).
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In case a config script is called from a relative path, the $(dirname
$0) would return a relative path too.
Those paths are usually parts of includes or libraries search
directories, and the packagfes buildsystems may chdir() anywhere, and
thus the relative path will no longer be valid. For example:
$ ./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/net-snmp-config --cflags
[...] -I./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/../../.././bin/../powerpc-buildroot-linux-uclibc/sysroot/usr/include/libnl3 [...]
Canonicalise the path to be sure we use absolute paths.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adjusts the logic in pkg-generic.mk that tweaks the
*-config shell scripts installed by various libraries to make it
compatible with per-package directories.
This requires two fixes:
- replacing $(STAGING_DIR) with a relative path from the config script
to the staging directory, rather than using an absolute path of the
staging directory.
Without this, a *-config script provided by package A, but called
from package B per-package directory will return paths from package A
per-package directory:
$ ./output/per-package/mcrypt/host/usr/<tuple>/sysroot/usr/bin/libmcrypt-config --libs
-L..../output/per-package/libmcrypt/host/usr/<tuple>/sysroot/usr/lib/
The libmcrypt-config script is installed by the libmcrypt package,
and mcrypt is a package that depends on libmcrypt. When we call the
libmcrypt-config script from the mcrypt per-package directory, it
returns a -L flag that points to the libmcrypt per-package
directory.
One might say: but this is OK, since the sysroot of the libmcrypt
per-package directory also contains the libmcrypt library. This is
true, but we encounter a more subtle issue: because -L paths are
considered before standard paths, ld ends up finding libc.so in the
libmcrypt per-package directory. This libc.so file is a linker
script that looks like this:
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a AS_NEEDED ( /lib/ld-linux.so.3 ) )
Normally, thanks to ld sysroot awareness, /lib/libc.so.6 in this
script is re-interpreted according to the sysroot. But in this
case, the library is *outside* the compiler sysroot. Remember: we
are using the compiler/linker from the "mcrypt" per-package
directory, but we found "libc.so.6" in the "libmcrypt" per-package
directory.
This causes the linker to really use the /lib/libc.so.6 from the
host machine, obvisouly leading to a build failure such as:
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/libc.so.6
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /usr/lib/libc_nonshared.a
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/ld-linux-nios2.so.1
- Some *-config scripts, such as the apr-1-config script, contain
references to host tools:
CC=".../output/per-package/apr/hosr/bin/arm-linux-gcc"
CCP=".../output/per-package/apr/hosr/bin/arm-linux-cpp"
We also want to replace those with proper relative paths. To
achieve this, we need to also replace $(HOST_DIR) with a relative
path. Since $(STAGING_DIR) is inside $(HOST_DIR), the first
replacement of $(STAGING_DIR) by @STAGING_DIR@ is no longer needed:
replacing $(HOST_DIR) by @HOST_DIR@ is sufficient. We still need to
replace @STAGING_DIR@ by the proper path though, as we introduce
@STAGING_DIR@ references in exec_prefix and prefix variables, as
well as -I and -L flags.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.
For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.
This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.
Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.
Thanks to those changes, the global "dirs" target is entirely removed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Recently, some hash mismatch have been reported, both by users as well
as autobuilder failures, about tarballs generated from git repositories.
This turned out to be caused by users having the 'gzip' command somehow
aliased to 'pigz' (which stand for: parallel implementation of gzip,
which takes advantage of multi-processor system to parallelise the
compression).
Unfortunately, the output of pigz-compressed archives differ from that
of gzip (even though they *are* valid gzip-compressed streams).
Add a dependency check that ensures that gzip is not pigz. If that is
the case, define a conditional dependency to host-gzip, that is used as
a download dependency for packages that will generate compressed files,
i.e. cvs, git, and svn.
Fixes:
http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Marcin Niestrój <m.niestroj@grinn-global.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The commands like "make show-build-order" or "make
<package>-show-build-order" show the build order and then print
"make[1]: Nothing to be done for 'show-build-order'" to stdout. It
pollutes output. Technically this message is true but it's not true
for user because he gets an information.
The <package>-show-build-order targets use $(info) for package name
printing. The make utility doesn't consider the internal directive as
a command so it think that it's "Nothing to be done". The patch adds
the empty command to <package>-show-build-order to inform make utility
that taget makes some real actions.
Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: invert $(info) and @:, as suggested by Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Three of our download backends need a host tar that can generate
reproducible archives: cvs, git, and svn. The other two, bzr and hg,
use their internal implementation.
So, for those three that need it, and a dependency on host-tar when the
system tar is not appropriate.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For some packages, we may need to have a certain set of host-tools built
before the download of said packages are attempted. For example, when
the system tar is not suitable, we will want to build our own tar before
we attempt a git download (because we generate a tarball in the git
backend).
Mimick the _EXTRACT_DEPENDENCIES, and introduce _DOWNLOAD_DEPENDENCIES.
As for _EXTRACT_DEPENDENCIES, we do not document _DOWNLOAD_DEPENDENCIES,
on the assumption that it is mostly for internal use.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the pkg-inner macros, all variables, but the positional arguments,
must be $$-prefixed, so that they are expanded only when the macro is
evaluated in each package, not when the macro is parsed.
It is to be noted, though, that the current code, even though
incorrect by the above rules, seemed to work. However, the upcoming
addition of download dependencies, mimicking that code, would not work
unless it was $$-prefixed.
So, for consistency sake, and for correctness sake, let's always use
the $$-prefix in the inner macro.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This adds one column to the legal-info manifest table. It contains the
dependencies of the given package and their licenses. This information
is useful when assessing license compatibility of the packages and
their libraries.
An example of the content of the new column for the MPD package is
shown below:
"alsa-lib [LGPL-2.1+ (library), GPL-2.0+ (aserver)] boost
[BSL-1.0] libid3tag [GPL-2.0+] libmad [GPL-2.0+] libogg
[BSD-3-Clause] libvorbis [BSD-3-Clause] libzlib [Zlib]
skeleton-init-common [unknown] skeleton-init-sysv [unknown] sqlite
[Public domain] toolchain-external-linaro-arm [unknown]"
[Credits to Yann E. MORIN <yann.morin.1998@free.fr> for suggesting a
few simplifications.]
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The last parameter {HOST|TARGET} is now first. With this change,
adding new columns to the legal manifest file (as in the next commit)
will be slightly easier to review.
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When a package has a version selection (e.g. Qt5), the licensing terms
may be different across versions, but lie in similarly named files (e.g.
'LICENSE').
However, when we check a file, all the hashes for it must match. So, we
can't have the hashes for two different content of the same file. We
overcame that limitation in the legal-license-file macro, which checks
whether a package has a .hash file in a versioned subdir.
For consistency, we would like to also store the source hashes in that
per-version subdir.
Rather than reconstruct the path to the hash file everywhere we need it,
add a variable that points to it.
Existing users will be converted over in followup patches.
Note: the check for a missing hash file is done in the check-hash helper
script, so this variable must always yield a filename, even of a missing
file, thus we do not use $(wildcard...) to resolve the hash file path;
we use $(wildcard...) only to check if the versioned .hash file exists.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Baruch Siach <baruch@tkos.co.il>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, the timestamps that we keep in build-time.log use a
second-level precision. However, as we are going to introduce a new
type of graph to draw the time line of a build, this precision is
going to be insufficient, as a number of steps are so short that they
are not even one second long, and generally the rounding to the second
gives a not so great looking graph.
Therefore, we add to the timestamps the nanoseconds using the %N date
specifier. A milli-second precision would have been sufficient, but %N
is all what date(1) provides at the sub-second level.
Since this is changing the format of the build-time.log file, this
commit adjusts the support/scripts/graph-build-time script
accordingly, to account for the floating point numbers that we have as
timestamps.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the site method is explicitly set to a SCM other than
git, the main download is correctly excluded from being hash-checked.
But when the site method is inferred from the site uri, the download
from a SCM other than git is wrongly being hash-checked.
Fix this by moving the code that excludes SCM methods from hash-check
below the code that infers the site method.
Currently there is no package in the tree that uses inferred site
method, and that is why the autobuilder didn't caught this. We had
packages using inferred site method in the past, the last one was
'expect', but since they didn't have a hash file (for a license or
extra-download for example) the build didn't error out.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 'local' site method is easily confused with the 'file' site method,
making people create packages like this:
FOO_SITE_METHOD = local
FOO_SOURCE = foo.tar.gz
$(eval $(generic-package))
Due to the intricacies of the generic package infra, this does not
cause an error; instead, the foo.tar.gz tarball that happens to be
present in the download directory will be used. This behaviour differs
greatly from what is specified in the manual.
Instead, error out immediately if a package specifies the 'local' site
method but does not specify a _SITE.
We check for _OVERRIDE_SRCDIR rather than checking for _SITE, just
after _OVERRIDE_SRCDIR has been set to _SITE. Indeed, a package that
sets _OVERRIDE_SRCDIR but not _SITE currently works correctly. There is
no reason to make it fail.
See also
https://stackoverflow.com/questions/50364655/including-patches-to-build-root
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Also call the step hooks from the three steps they are currently not
called in:
- download,
- actual download (when main archive is not the real source, like
external toolchains),
- rsync (for local or override-srcdir).
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit ea55e1323 (core/pkg-infra: don't enforce site-method for extra
downloads) forgot to account for those packages that have nothing to
download, like the skeleton, or like virtual packages...
The side effect is that the message "foo Downloading" is thus
displayed when it should not be.
Fix that.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The site method is stored in FOO_SITE_METHOD, not in FOO_SITE.
Fixes:
http://autobuild.buildroot.net/results/131/13196dd779bc9e3b172c74851546dd4c4752aa02/
[Peter: add autobuilder reference]
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The site method only ever applies to the main download, while extra
downloads are always to be fetched with wget.
However, the site method is prepended to the URL from within the
DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
which is called for each download of a package, thus effectively
prepending the site method to all downloads, even the extra ones (and
the patches).
We fix that by prepending the site method from within the
generic-package infra, so that it only applies to the main download.
For that, we move the main _SOURCE out of the foreach loop, so that
we can prepend the site-method to it, without impacting the other
downloads.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the main download if from an SCM, we exclude all
downloads from being hash-checked, on the assumption that we don't
have hashes for downloads from an SCM.
However, the exclusion is computed on the DOWNLOAD macro, which is
called for each download of a package, thus effectively disabling
hash checks for extra downloads, even though those are only ever
download with wget.
What we really wanted to do, in fact, was to exclude just the main
download.
We fix that by appending the main source file to the global list of
excluded files, from within the generic-package infra itself.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When rebuilding a package, simply appending the package's file list to
the global list means that the package list grows for every rebuild, as
does the time taken to check for files installed by multiple packages.
Furthermore, we get false positives where a file is reported as being
installed by multiple copies of the same package.
With this approach we may end up with orphaned files in the target
filesystem if a package that has been updated and rebuilt no longer
installs the same set of files, but we know that only a clean build will
produce reliable results. In fact it may be helpful to identify these
orphaned files as evidence that the build is not clean.
Signed-off-by: John Keeping <john@metanate.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Update the documentation accordingly.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998@free.fr: slight rephrasing in error message, update manual]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes:
http://autobuild.buildroot.net/results/b4a/b4af0de4ae9630ccbe7890f69047f216f2ff5119/
With the change to the DOWNLOAD macro, packages using FOO_ACTUAL_SOURCE_TARBALL fails:
>>> skeleton-init-common Collecting legal info
sourceryg++-2017.05-4-nios2-linux-gnu.src.tar.bz2: OK (md5: 529a7fecf33d0d113a446413b9d1e173)
sourceryg++-2017.05-4-nios2-linux-gnu.src.tar.bz2: OK (sha256: 6e65878d0453708ee19098d3d68985bda244938d35999f3859915a2f5574fa08)
/bin/bash: line 1: @mkdir: command not found
package/pkg-generic.mk:148: recipe for target '/accts/mlweber1/rclinux/rc-buildroot-test/scripts/instance-1/output/build/toolchain-external-codesourcery-niosII-2017.05-4/.stamp_actual_downloaded' failed
Which is caused by the continuation character '\'. This has been present
since the make target was introduced in commit eace9d6133
(core/legal-info: ensure legal-info works in off-line mode). It isn't clear
to me why it was done like that, but it fails with the DOWNLOAD macro
rework, so drop it.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This per package variable can be used to specify the download
subdirectory used by that package.
The use case here is for example linux-headers and linux, which share
the same sources (because they are the same upstream project), so we
don't want to download twice the kernel, nor store it multiple times
either.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With all the previous changes, we are now ready to add a subdirectory to
the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,
which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name
(e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the
newly created subdirectory.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Let the infrastructure use the already existing variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit adds the support for <pkg>-show-recursive-depends and
<pkg>-show-recursive-rdepends which respectively show the list of all
dependencies or reverse dependencies for a given package. The existing
show-depends and show-rdepends only show the first-level dependencies,
while show-recursive-depends and show-recursive-rdepends show
recursively the dependencies.
It is worth mentioning that while show-recursive-depends really shows
all dependencies, show-recursive-rdepends is a bit limited because the
reverse dependencies of host packages are not properly accounted
for. But that's a limitation that already exists in show-rdepends, and
that cannot easily be solved.
Signed-off-by: George Redivo <george.redivo@datacom.ind.br>
[Thomas:
- split from the patch that was also changing graph-depends
- rename show-rrdepends to show-recursive-rdepends
- add show-recursive-depends
- don't create GRAPHS_DIR.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to
skip copying parts of source trees unneeded for building. For example,
when developing WebKitGTK+, it's handy to skip copying all the tests and
other build directories, which are huge:
WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
--exclude JSTests --exclude ManualTests \
--exclude PerformanceTests --exclude WebDriverTests \
--exclude WebKitBuild --exclude WebKitLibraries \
--exclude WebKit.xcworkspace --exclude Websites \
--exclude Examples
This saves a good chunk of time when rsync is used for the first time to
copy the source tree over before building.
Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
[Arnout: move documentation to the end of the section]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In current Buildroot, clashes occur between the variables _NAME and
_BASE_NAME for two packages called foo and foo-base, i.e.
Package foo:
FOO_NAME = foo
FOO_BASE_NAME = foo-1.2.3
Package foo-base:
FOO_BASE_NAME = foo-base
FOO_BASE_BASE_NAME = foo-base-4.5.6
where variable FOO_BASE_NAME is clashing between these two packages.
Specific cases where this clash is already existing are:
- alljoyn-base
- alljoyn-tcl-base
- perl-xml-sax-base
The problem is generic and can occur for a number of variables in Buildroot.
A non-exhaustive list:
<pkg>_BASE and <pkg>_BASE_NAME
<pkg>_BASE_NAME and <pkg>_RAW_BASE_NAME
<pkg>_DIR and <pkg>_DL_DIR
<pkg>_VERSION and <pkg>_DL_VERSION
<pkg>_SOURCE and <pkg>_TARGET_SOURCE
<pkg>_INSTALL_IMAGES and <pkg>_TARGET_INSTALL_IMAGES (same for _STAGING and _TARGET)
<pkg>_LICENSE_FILES and <pkg>_MANIFEST_LICENSE_FILES
<pkg>_DEPENDENCIES and <pkg>_FINAL_DEPENDENCIES
One solution is to use another separator than '_' to separate the
package name from the rest of the variable name. For example, a double
underscore:
FOO__NAME
FOO__BASE_NAME
FOO_BASE__NAME
FOO_BASE__BASE_NAME
However, making that change for only this case means that the variable
naming is no longer consistent. And making the change for all variables has
a large impact, also on certain user scripts.
For now, keep it simple, and rename FOO_BASE_NAME into FOO_BASENAME, so that
the variables become:
FOO_NAME
FOO_BASENAME
FOO_BASE_NAME
FOO_BASE_BASENAME
For consistency, also adapt FOO_RAW_BASE_NAME. Since FOO_RAW_BASENAME would
still pose a conflict with a package called 'foo-raw', take the opportunity
to rename it into FOO_BASENAME_RAW instead, which does not pose a conflict
as we have no variable called FOO_RAW.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Sam Voss <sam.voss@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that the pre-fs ones are run on a transient copy of target/, the
post-fs hooks are no longer needed because we no longer need to restore
the target/ directory as it is only a internal copy.
Remove support for the post-fs hooks, and update the sole package using
them.
We do not add a legacy check because this was mostly a purely-internal
detail that was never really exposed nor documented.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Like we do for packages with the PKG variable, set ROOTFS to contain the
upper-case name of the rootfs currently being generated.
This will be useful in later patches, when we need more per-rootfs
variables, like a per-rootfs TARGET_DIR for example.
In Makefiles, per-rule variables trickle down the dependency chain, to
all dependencies of that rule, so we have to stop ROOTFS as soon as
we're not in a rootfs. This means we have to stop it at target-finalize
(which is a dependency of all filesystems), and for each package
individually, since some packages (host or target) can be direct
dependencies of filesystems as well.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that DEPENDENCIES_HOST_PREREQ is no longer used anywhere, we can
kill it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit moves the host-fakedate dependency handling from
DEPENDENCIES_HOST_PREREQ to a proper regular dependency handled by the
package infrastructure.
host-fakedate is added as dependency to all packages, except
host-skeleton, because we depend on it.
In addition, we make sure that host-fakedate does not grow a
dependency on host-{tar,xz,lzip,ccache} to avoid circular
dependencies. host-fakedate does not need any extraction tool and does
not need to build C/C++ code (the source code is just a shell script
available in Buildroot).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This moves the host-ccache dependency handling from
DEPENDENCIES_HOST_PREREQ to a proper package dependency. When
BR2_CCACHE=y, we add host-ccache as a regular dependency of all
packages except:
- The extractor packages host-tar, host-xz and host-lzip
- host-ccache itself
- host-skeleton, because all packages depend on it
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This moves the host-lzip dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.
To achieve that, check-host-lzip.mk fills in the
BR2_LZIP_HOST_DEPENDENCY with host-lzip if building a host-lzip is
needed. The name BR2_LZIP_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.
The BR2_LZIP_HOST_DEPENDENCY is added to all packages, except:
- host-lzip, because we would otherwise depend on ourself.
- host-tar, because lzip itself is delivered as a tarball, so we need
to have host-lzip depend on host-tar, and not host-tar depend on
host-lzip
- host-skeleton, because we need to have host-lzip depend on
host-skeleton, and not the opposite.
We also mutually exclude host-lzip and host-xz from dependending on
each other, to avoid a circular dependency.
In addition, we modify lzip.mk to explicitly build host-lzip without
ccache. We generally took the approach of building host-ccache *after*
all the extractors have been built.
[Peter: fix s/host-tar/host-lzip/ typo, fix s/xz/lzip/ typo]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This moves the host-xz dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.
To achieve that, check-host-xz.mk fills in the
BR2_XZ_HOST_DEPENDENCY with host-xz if building a host-xz is
needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.
The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:
- host-xz, because we would otherwise depend on ourself.
- host-tar, because xz itself is delivered as a tarball, so we need
to have host-xz depend on host-tar, and not host-tar depend on
host-xz
- host-skeleton, because we need to have host-xz depend on
host-skeleton, and not the opposite.
In addition, we modify xz.mk to explicitly build host-xz without
ccache. We generally took the approach of building host-ccache *after*
all the extractors have been built.
[Peter: fix s/host-tar/host-xz/ typo]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This moves the host-tar dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.
To achieve that, check-host-tar.mk fills in the
BR2_TAR_HOST_DEPENDENCY variable with host-tar if building a host-tar
is needed. The name BR2_TAR_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.
The BR2_TAR_HOST_DEPENDENCY is added to all packages, except host-tar
itself (obviously) and host-skeleton, because we depend on
host-skeleton to install host-tar properly in HOST_DIR.
In addition, we modify tar.mk to explicitly build host-tar without
ccache: since ccache source code is available as a tarball, ccache
will obviously depend on host-tar if the system tar is insufficient.
Finally, to make things really clean, we also add
$(BR2_TAR_HOST_DEPENDENCY) to the dependencies of the tar filesystem
format, since it requires tar, so we'd better make sure we have a
suitable tar.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Extract dependencies are dependencies that must be ready before the
extract step of a package, i.e for tools that are needed to extract
packages themselves. Current examples of such tools are host-tar,
host-lzip and host-xz.
They are currently handled through DEPENDENCIES_HOST_PREREQ. However,
this mechanism has a number of drawbacks:
- First and foremost, because host-tar/host-lzip/host-xz are not
listed in the dependencies of packages, the package infrastructure
does not know it should rsync them in the context of per-package
SDK.
- Second, there is no dependency handling *between* them. I.e, we
have no mechanism that says host-tar should be built before
host-lzip, while it is in fact the case: if you need to build
host-lzip, you need to extract a tarball, so you may need host-tar
if your system tarball is not capable enough.
For those reasons, it makes sense to add explicit support for "extract
dependencies" in the package infrastructure, through the
<pkg>_EXTRACT_DEPENDENCIES variable. It is unlikely this variable will
ever be used by a package .mk file, but it will be used internally by
the package infrastructure.
[Peter: fix typo in manual]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As part of the per-package SDK work, we want to avoid having logic
that installs files to the global HOST_DIR, and instead do it inside
packages. One thing that gets installed to the global HOST_DIR is the
minimal "skeleton" that we create in host:
- the "usr" symbolic link for backward compatibility
- the "lib" directory, and its lib64 or lib32 symbolic links
This commit moves this logic to a new host-skeleton package, and makes
all packages (except itself) depend on it.
While at it, use $(Q) instead of @ in the HOST_SKELETON_INSTALL_CMDS.
[Peter: drop host-patchelf reference in commit message]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As part of the build, we run some instrumentation hooks to gather
statistics about the usage of the target/, staging/ and host/
directories, so that we can generate reports for the user, that
shows:
- for each file, what package installed it,
- for each package,the size that it installed.
In so doing, we run a double md5 pass on all files of the affected
directories (before/after installation). These passes were mostly invisible
when we were only scanning target/, but has greatly increased in time now
that we also scan staging/ and host/ (but only in the corresponding _CMDS,
of course).
This md5 was mostly aimed at catching packages that would "cheat" with
mtime/atime/ctime somehow. They can't really cheat on md5, though [0].
Timings however speak for themselves, with this defconfig (slightly
biggish-but-still-manageable build) [1].
host/ 20965 files 1.2GiB
staging/ 4715 files 333MiB
target/ 1801 files 44MiB
All instrumentation steps, using md5: 19min 27s
All instrumentation steps, using mtime: 14min 45s
No instrumentation step at all: 14min 31s
So, using mtime is an almost-5min improvement, i.e. about 25% faster,
while removing all instrumentation steps does not gain that much more...
So, we switch to using mtime, because in the end that's still good-enough
for our use-case: generating some graphs. It is not mission-critical, and
if a graph is slightly off, that's not a biggy. It can anyway be attributed
to a broken package's buildsystem, which should get fixed.
However, we lose the ability to track directories. Non-empty directories
can be tracked back by a bit of scripting, but empty directories are
simply not caught. If we were to also look for directories using mtime,
we would catch parents of installed files:
- /foo/bar/ exists
- a package installs /foo/bar/buz
- mtime of /foo/bar/ is changed to account for the new file in it.
So we do not track directories at all, and we lose empty directories.
The existing tracking was mostly happenstance, with the original
submission and comments not really accounting for a real use-case.
Now, we also change the way we handle symlinks. Previously, we would
hash the file pointed to by the symlink. Now, we only look at the mtime
of the symlink itself, which still detects modifications.
Eventually, this also means that we now no longer need to establish a
list before the install step; we can now simply run after the install
step, finding any files newer than the build stamp.
[0] Yeah, md5 is very weak, but we're not guarding against malicious
attacks, just about careless modifications.
[1] defconfig used for tests:
BR2_arm=y
BR2_cortex_a7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_INIT_SYSTEMD=y
BR2_PACKAGE_MESA3D=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_ETNAVIV=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_VC4=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_VIRGL=y
BR2_PACKAGE_MESA3D_DRI_DRIVER_SWRAST=y
BR2_PACKAGE_MESA3D_OSMESA=y
BR2_PACKAGE_MESA3D_OPENGL_ES=y
BR2_PACKAGE_SYSTEMD_JOURNAL_GATEWAY=y
BR2_PACKAGE_SYSTEMD_BACKLIGHT=y
BR2_PACKAGE_SYSTEMD_BINFMT=y
BR2_PACKAGE_SYSTEMD_COREDUMP=y
BR2_PACKAGE_SYSTEMD_FIRSTBOOT=y
BR2_PACKAGE_SYSTEMD_HIBERNATE=y
BR2_PACKAGE_SYSTEMD_IMPORTD=y
BR2_PACKAGE_SYSTEMD_LOCALED=y
BR2_PACKAGE_SYSTEMD_LOGIND=y
BR2_PACKAGE_SYSTEMD_MACHINED=y
BR2_PACKAGE_SYSTEMD_POLKIT=y
BR2_PACKAGE_SYSTEMD_QUOTACHECK=y
BR2_PACKAGE_SYSTEMD_RANDOMSEED=y
BR2_PACKAGE_SYSTEMD_RFKILL=y
BR2_PACKAGE_SYSTEMD_SMACK_SUPPORT=y
BR2_PACKAGE_SYSTEMD_SYSUSERS=y
BR2_PACKAGE_SYSTEMD_VCONSOLE=y
[Peter: tweak commit message, use find -type l]
Reported-by: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Trent Piepho <tpiepho@impinj.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some packages (mostly, out-of-tree) may want to install binary blobs for
another architecture, outside the locations we currently exclude, like
in /opt or whatever...
Add support in check-bin-arch to accept any arbitrary location, that
individual package can each request to excude from the check, when they
are installed.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we store the list of files installed in target/ and associate
each of them to the package that installed it.
However, we sometimes may need to know what package installed which file
in staging/, for example to debug header collision, or in host/, to
debug what package installed what host tool.
Enhance the step instrumentation to also generate the list for staging/
and host/.
We maintain backward compatibility, for external scripts that wanted to
parse the previously existing list, by not renaming the target-related
package list. Only the staging- and host-related lists are named after
staging and host.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Thomas: fix missing word in .mk comment.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
There is no need to redirect again and again for each new file added to
the list; we can just redirect once and for all.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To compute the list of files added by a package, we first store the list
of files before the install, do the install, list the files after the
install, and finally compare the two lists. The two lists are stored in
dot-files, hidden in the package's build dir.
We currently keep those two files, and only list the files installed in
target/
In followup patches, we'll also list files installed in staging/ as well
as files installed in host/.
Rather than add even more internal, hidden files in the package build
dir, we'll just re-use the same two temporary files to store the before
and after lists.
So, remove them after the comparison is done.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
