Currently, the wording in the manual instructs the user to generate a
tarball from "the contents of the +output/host+ directory".
This is pretty confusing, because taken literally, this would amount to
running a command like:
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when
extracted, it creates multiple files in the current directory.
What one really wants to do, is create a tarball of the host/ directory,
with something like:
tar cf my-sdk.tar -C output host/
However, this is not much better, because the top-most directory would
have a very common name, host/, which is pretty easy to get conflict
with when it gets extracted.
So, we fix that mess by giving the top-most directory a recognisable
name, based on the target tuple, which we also use as the name of the
archive (suffixed with the usual +.tar.gz+.) We offer the user the
possibility to override that default by specifying the +BR2_SDK_PREFIX+
variable on the command line.
Since this is an output file, we place it in the images/ directory.
As some users expressed a very strong feeling that they do not want to
generate a tarball at all, and that doing so would badly hurt their
workflows [1], we actually prepare the SDK as was previously done, but
under the new, intermediate rule 'prepare-sdk'. The existing 'sdk' rule
obviously depend on that before generating the tarball.
We choose to make the existing rule to generate the tarball, and
introduce a new rule to just prepare the SDK, rather than keep the
existing rule as-is and introduce a new one to generate the tarball,
because it makes sense to have the simplest rule do the correct thing,
leaving advanced, power users use the longest command. If someone
already had a wrapper that called 'sdk' and expected just the host
directory to be prepared, then this is not broken; it just takes a bit
longer (gzip is pretty fast).
Update the manual accordingly.
[0] https://en.wikipedia.org/wiki/Tar_(computing)#Tarbomb
[1] http://lists.busybox.net/pipermail/buildroot/2018-June/thread.html#223377
and some messages in the ensuing thread...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Wolfgang Grandegger <wg@grandegger.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Stefan Becker <chemobejk@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: "Yann E. MORIN" <<a href="mailto:yann.morin.1998@free.fr" target="_blank">yann.morin.1998@free.fr</a>><br>
Reviewed-by: Stefan Becker <chemobejk@gmail.com>
Signed-off-by: "Yann E. MORIN" <<a href="mailto:yann.morin.1998@free.fr" target="_blank">yann.morin.1998@free.fr</a>><br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The snippet of code that runs a check-package on all
.mk/.hash/Config.in files is currently only available within
.gitlab-ci.yml, and isn't immediately and easily usable by Buildroot
users. In order to simplify this, this commit introduces a top-level
"check-package" make target that implements the same logic. The
.gitlab-ci.yml file is changed to use "make check-package".
Since this target is oriented towards Buildroot developers, we
intentionally do not clutter the already noisy "make help" text with
this additional make target.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The help text says that BR2_DEFCONFIG will be used as input, but a
BR2_DEFCONFIG specified in the existing .config file will *not* be
used. So say explicitly that it must be specified on the command line.
Note that both "BR2_DEFCONFIG=... make defconfig" and
"make defconfig BR2_DEFCONFIG=..." will work.
While we're at it, add a semicolon to separate the two statements.
Note that this overflows the help text beyond 80 characters, but that
is already the case in many other lines.
Reported-by: Anisse Astier <anisse@astier.eu>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The systemd ConditionNeedsUpdate option is useful when offline updates
of the vendor operating system resources in /usr require updating of
/etc or /var on the next following boot.
Two examples of services making use of this option are
systemd-hwdb-update.service and systemd-sysusers.service.
ConditionNeedsUpdate=/etc will be true if the mtime of /etc/.updated
is older than the mtime of /usr. After services conditional on
ConditionNeedsUpdate have run, systemd-update-done.service will
synch the mtime of /usr to /etc/.updated so that the condition will
be false on subsequent boots.
For systems with writable /usr partitions where updates are done to
the running system, the update program will touch /usr as a final step.
But with Buildroot, where updates are often done by dumping a new
image onto the device, and where /usr is on a filesystem mounted
read-only, touching /usr as part of the update process is not practical.
Instead, it should be done a build time.
For testers, please note that systemd-update-done in v234 added a
regression where the mtime of /etc/.updated is set to the current time
instead of the mtime or /usr. This will be fixed in v239.
For more details, see:
http://0pointer.de/public/systemd-man/systemd.unit.htmlhttp://0pointer.de/public/systemd-man/systemd-update-done.service.html
Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The Blackfin architecture has for a long time been complicated to
maintain, with poor support in upstream binutils/gcc. As of April
2018, the Blackfin architecture has been dropped from the upstream
Linux kernel. Also, the Analog Device engineer who used to be in touch
with the Buildroot community also privately said we should drop the
support for this architecture, which Analog Devices is no longer
using, promoting and maintaining.
The BR2_BINFMT_FLAT_SEP_DATA option becomes unselectable, it will be
removed in a future commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For reproducible builds, SOURCE_DATE_EPOCH will be set to the git commit
date if it is not defined in the environment, but this was done by
explicitly using $(TOPDIR)/.git as the git repository, which would not
give the expected result if Buildroot had been put into a subdirectory
of another repository.
This commit removes that restriction, meaning that the default date will
now be the date of the git commit that contains Makefile, regardless of
what level above Makefile the repository is at. This works because the
current directory when the 'git log' command is executed will always be
the directory containing Makefile (it must be, since TOPDIR is set from
CURDIR).
In general this should be a sensible default, and in cases where a
different date is required SOURCE_DATE_EPOCH can be defined in the
environment before invoking make.
Signed-off-by: James Byrne <james.byrne@origamienergy.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
If SOURCE_DATE_EPOCH is not defined it was given a definition that
caused 'git log' to be executed each time the variable is referenced,
which is not very efficient given that the answer cannot change.
This commit moves the definition of BR2_VERSION_GIT_EPOCH after the
inclusion of Makefile.in (so that GIT is defined) and makes it a
simply expanded variable so that it is only evaluated once.
Signed-off-by: James Byrne <james.byrne@origamienergy.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds the support for <pkg>-show-recursive-depends and
<pkg>-show-recursive-rdepends which respectively show the list of all
dependencies or reverse dependencies for a given package. The existing
show-depends and show-rdepends only show the first-level dependencies,
while show-recursive-depends and show-recursive-rdepends show
recursively the dependencies.
It is worth mentioning that while show-recursive-depends really shows
all dependencies, show-recursive-rdepends is a bit limited because the
reverse dependencies of host packages are not properly accounted
for. But that's a limitation that already exists in show-rdepends, and
that cannot easily be solved.
Signed-off-by: George Redivo <george.redivo@datacom.ind.br>
[Thomas:
- split from the patch that was also changing graph-depends
- rename show-rrdepends to show-recursive-rdepends
- add show-recursive-depends
- don't create GRAPHS_DIR.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This was present in Yann's original patch, but got dropped when I rebased
commit 7e9870ce32 (core: introduce intermediate BASE_TARGET_DIR variable) to
fix the Makefile conflicts.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Each of the intermediate, per-rootfs target directories, as well as the
intermediate tarball, can take quite some place, and is mostly a
duplication of what's already in target/. The only delta, if any, would
be the tweaks made by the filesystem image generations, but those tweaks
are most probably only meaningful when seen as root.
We normally do not remove intermediate files, but those can be quite
large, and are not directly usable by, nor accessible to the user.
So, get rid of them once the filesystem has been generated.
This does not need to be done in fakeroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
... which for now still points to the base target directory, but this is
a step forward.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Like we do for packages with the PKG variable, set ROOTFS to contain the
upper-case name of the rootfs currently being generated.
This will be useful in later patches, when we need more per-rootfs
variables, like a per-rootfs TARGET_DIR for example.
In Makefiles, per-rule variables trickle down the dependency chain, to
all dependencies of that rule, so we have to stop ROOTFS as soon as
we're not in a rootfs. This means we have to stop it at target-finalize
(which is a dependency of all filesystems), and for each package
individually, since some packages (host or target) can be direct
dependencies of filesystems as well.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This new BASE_TARGET_DIR variable is set in stone to point to the real
location where packages will be installed. Its name is modelled after
its definition: it is located in $(BASE_DIR), and it is named 'target/',
hence BASE_TARGET_DIR.
The already-existing TARGET_DIR variable now simply points to the same
location, except that it is recursively expanded, so that we can later
change it depending on the context.
All locations that really need to reference the existing target/
directory, are changed to use BASE_TARGET_DIR; surprinsigly enough, they
all seem to be located in the main Makefile. :-) The rest is left with
using good-old TARGET_DIR.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
If BR2_HOST_DIR is not the default, it can be difficult to find the
host directory (i.e., HOST_DIR always has to be passed explicitly in
addition to the output directory). For example, the Eclipse plugin
assumes that HOST_DIR=BASE_DIR/host.
Create a symlink from $(BASE_DIR)/host to $(HOST_DIR) if it is not the
default. Also remove it in the clean target.
When BR2_HOST_DIR is the default, HOST_DIR_SYMLINK will be empty so
there will be no additional dependency to dirs and nothing to remove
in clean.
Fixes https://bugs.busybox.net/show_bug.cgi?id=10151
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
HOST_DIR is defined twice: once to its default value before .config is
included, and once more to BR2_HOST_DIR after .config is included.
However, the rule that defines the mkdir for HOST_DIR comes between
these two, so it will always use the default definition. Therefore,
if a non-default BR2_HOST_DIR is used, there will be no rule to create
that directory, while the dirs target depends on it.
This happens to work at the moment, because in the dirs target,
$(STAGING_DIR) comes before $(HOST_DIR), so $(HOST_DIR) will be created
implicitly. However, this will fail in top-level parallel builds where
both will be created in parallel.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
After commit 6729050f3a nothing creates
$(HOST_DIR)/share/buildroot anymore, causing sdk to fail with:
/bin/bash: .../output/host/share/buildroot/sdk-location: No such file or directory
Add creation of that directory to the "sdk" build steps itself.
Signed-off-by: Stefan Becker <chemobejk@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that DEPENDENCIES_HOST_PREREQ is no longer used anywhere, we can
kill it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit moves the host-fakedate dependency handling from
DEPENDENCIES_HOST_PREREQ to a proper regular dependency handled by the
package infrastructure.
host-fakedate is added as dependency to all packages, except
host-skeleton, because we depend on it.
In addition, we make sure that host-fakedate does not grow a
dependency on host-{tar,xz,lzip,ccache} to avoid circular
dependencies. host-fakedate does not need any extraction tool and does
not need to build C/C++ code (the source code is just a shell script
available in Buildroot).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As part of the per-package SDK work, we want to avoid having logic
that installs files to the global HOST_DIR, and instead do it inside
packages. One thing that gets installed to the global HOST_DIR is the
minimal "skeleton" that we create in host:
- the "usr" symbolic link for backward compatibility
- the "lib" directory, and its lib64 or lib32 symbolic links
This commit moves this logic to a new host-skeleton package, and makes
all packages (except itself) depend on it.
While at it, use $(Q) instead of @ in the HOST_SKELETON_INSTALL_CMDS.
[Peter: drop host-patchelf reference in commit message]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some packages really want to use an UTF-8 locale, or they break.
However, there is no guarantee that any given locale is available on a
system. For example,, while most mainstream distros (Debian and
derivatives, Fedora...) do have the generic, language-agnostic C.UTF-8
locale, Gentoo does not provide it.
So, find the first UTF-8 locale available on the system, and take any
that is available. We however do favour using the user-set current
locale, then using the language-agnostic C.UTF-8, and eventually any
random UTF-8 locale.
Note: we only need to enforce LC_ALL, because setting it implies
everything else:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_02
"""
1. If the LC_ALL environment variable is defined and is not null,
the value of LC_ALL shall be used.
"""
[Peter: use same regexp as in dependencies.sh]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
It is recommended that vendor trees store OS release information
in /usr/lib/os-release and that /etc/os-release should be a relative
symlink to /usr/lib/os-release.
For more details, see:
http://0pointer.de/public/systemd-man/os-release.html
[Peter: don't hide command, simplify ln invocation]
Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This merges the next branch accumulated during the 2017.11 release
cycle back into the master branch.
A few conflicts had to be resolved:
- In the DEVELOPERS file, because Fabrice Fontaine was added as a
developer for libupnp in master, and for libupnp18 in
next. Resolution is simple: add him for both.
- linux/Config.in, because we updated the 4.13.x release used by
default in master, while we moved to 4.14 in next. Resolution: use
4.14.
- package/libupnp/libupnp.hash: a hash for the license file was added
in master, while the package was bumped into next. Resolution: keep
the hash for the license file, and keep the hash for the newest
version of libupnp.
- package/linux-headers/Config.in.host: default version of the kernel
headers for 4.13 was bumped to the latest 4.13.x in master, but was
changed to 4.14 in next. Resolution: use 4.14.
- package/samba4/: samba was bumped to 4.6.11 in master for security
reasons, but was bumped to 4.7.3 in next. Resolution: keep 4.7.3.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, enabling more than one filesystem image will make
'show-targets' list a few host packages more than once.
This is because all filesystem images add the same set of
host-packages to their dependencies, which are then added as-is
to the package list.
Thus, host-fakeroot, host-makedevs and, if needed, host-mkpasswd will
appear as many times as there are filesystem images enabled.
Fix that by sorting the package list, thus eliminating duplicates from
that list. Also sort the rootfs list for good measure. Sort the two
separately, so that rootfses are last.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we do nothing about packages that touch the same file: given
a specific configuration, the result is reproducible (even though it
might not be what the user expected) because the build order is
guaranteed.
However, when we later introduce top-level parallel build, we will no
longer be able to guarantee a build order, by the mere way of it being
parallel. Reconciliating all those modified files will be impossible to
do automatically. The only way will be to refuse such situations.
As a preliminary step, introduce a helper script that detects files that
are being moified by two or more packages, and reports them and the
impacted packages, at the end of the build.
The list being reported at the end of the build will make it prominently
visible in autobuilder results, so we can assess the problem, if any.
Later on, calling that helper script can be done right after the package
installation step, to bail out early.
Thanks Arnout for the pythonist way to write default dictionaries! ;-)
Note: doing it in python rather than a shell script is impressively
faster: where the shell script takes ~1.2s on a minimalist build, the
python script only takes ~0.015s, that is about 80 times faster.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Peter Seiderer <ps.report@gmx.net>
[Thomas: rename script without .py extension.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The 'include' directive in GNU make supports wildcards, but their
expansion has no defined sort order (GLOB_NOSORT is passed to glob()).
Usually this doesn't matter. However, there is at least one case where
it does make a difference: toolchain/*/*.mk includes both the
definitions of the external toolchain packages and
pkg-toolchain-external.mk, but pkg-toolchain-external.mk must be
included first.
For predictability, use ordered 'include $(sort $(wildcard ...))'
instead of unordered direct 'include */*.mk' everywhere.
Fixes [1] reported by Petr Vorel:
make: *** No rule to make target 'toolchain-external-custom', needed by '.../build/toolchain-external/.stamp_configured'. Stop.
[1] http://lists.busybox.net/pipermail/buildroot/2017-November/206969.html
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Tested-by: Petr Vorel <petr.vorel@gmail.com>
[Arnout: also sort the one remaining include, of the external docs]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
SOURCE_DATE_EPOCH is currently forcibly set (to either the git commit
date, or the last release date).
However, the spec mandates that it should not be modified if already
set: https://reproducible-builds.org/specs/source-date-epoch/
Build systems MUST NOT overwrite this variable for child
processes to consume if it is already present.
Abide by the rule, and only set it if not already set.
This will allow users to pass it from an upper-layer buildsystem (e.g. a
jenkins or gitlab-ci job, for example), when they have a reson to do so.
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Reported-by: Einar Jón Gunnarsson <tolvupostur@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Einar Jón Gunnarsson <tolvupostur@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
That way packages included in that list like ccache will also be
regarded as a normal packages for targets like external-deps,
show-targets or legal-info
Signed-off-by: Alfredo Alvarez Fernandez <alfredo.alvarez_fernandez@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
