There is no need to break the "\n" sequence using "%sn". We can just
escape it. Note: the escaping backslash needs to be escaped too,
because the shell will process the string before printf gets to see it.
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that we can get the whole dependency tree from make, use it to
speed up things considerably.
So far, we had three functions to get the dependencies information:
get_depends(), get_rdepends(), and, somehow unrelated, get_version().
Because of the way %-show-{,r}depends works, getting the dependency tree
was expensive, the three functions all took a set of packages for which
to get the dependencies, in an attempt to limit the time it took to get
that tree, but we still had to call these functions iteratively, until
they returned no new dependency. This was pretty costly.
Now, getting the tree is much, much less costly, and we can get the
whole tree as cheaply as we previously got only the first-level
dependencies.
Furthermore, we can now also get the version information at the same
time, and that also brings in whether the package is virtual or not,
target or host.
So, we drop all three helper functions, and replace them with a single
one that returns all that information in one go: full dependency trees
(direct and reverse), per-package type, and per-package version.
Note: since commit 2d29fd96a (pkg-virtual: remove VERSION/SOURCE),
virtual packages are no longer reported as having a 'virtual' version,
so have since been displayed as regular packages in the graphs. Although
noone complained, this patch incidentally restores the initial
behaviour, and virtual packages are now correctly displayed as such
again.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We we simplify the dependency graph, we try to remove so-called
mandatory dependencies from each package, and for each mandatory that
was thus removed, reattach it to the root-package of the graph.
This was made so that mandatory dependencies (which are dependencies of
all packages, or at least of a lot of packages) do not clutter the
dependency graph, but that they are still shown in the graph, as
dependencies of the root package.
However, these mandatory dependencies are only _direct_ dependencies.
As such, it does not make sense to reattach a mandatory dependency when
doing a reverse graph. Worse, it can actually be incorrect.
For example, 'skeleton' is a mandatory dependency, and as such is
removed from all packages. But when doing a reverse graph, skeleton is
now in the dependency chain of, e.g. skeleton-init-none; it should then
not be removed.
In short: the notion of mandatory dependencies does not make sense in
the case of a reverse graph.
Consequently, skip over the mandatory dependency removal when doing a
reverse graph.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When host-gzip is needed, it is a mandatory dependency of all packages.
As such, drawing the dependency lines toward host-gzip would uselessly
clutter the graph.
So, like for the skeleton, host-skeleton, and host-tar, we cut the
dependency chains toward host-gzip.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When host-tar is needed, it is a mandatory dependency of all packages.
As such, drawing the dependency lines toward host-tar would uselessly
clutter the graph.
So, like for the skeleton and host-skeleton, we cut the dependency chains
toward host-tar.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
host-skeleton is a dependency of almost all packages, except a very few.
As such, it clutters the dependency graph uselessly.
Do with it as we do for the skeleton: cut the dependency chains.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some times, multiple dependency graphs for a set of packages (mostly
the application-level packages for the project) are included in reports
(e.g. delivery notes). Repeating the mandatory dependencies on all
those graphs is useless and clutters the important dependencies.
When we had only two such mandatory dependencies (toolchain, skeleton),
it was manageable to list them as manual exclusions:
-x toolchain -x skeleton
But we now have quite a few such dependencies, and it becomes a bit more
cumbersome to manage, not counting the ones we may add in the future.
Add an option to exclude all those mandatory dependencies, to generate
neat graphs.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The current graph-depends implementation filters out a number of
"mandatory" dependencies that all packages have: dependency on
"toolchain" and dependency on "skeleton".
Despite this filtering, in full graph dependencies, "toolchain" and
"skeleton" are still shown, because they are target packages, and
therefore appear in the result of "make show-targets". Thanks to this,
they will be visible as dependencies of the "ALL" node, which is the
root of the dependency tree.
However, as we are going to introduce host-skeleton as a "mandatory
dependency" to be filtered out, this is no longer going to work.
This commit adjusts the remove_extra_deps() function to ensure that
when a mandatory dependency is removed, this dependency exists between
the root of the dependency tree and the mandatory dependency.
This issue was noticed by Yann E. Morin, and this commit provides a
different implementation than what Yann proposed in
https://patchwork.ozlabs.org/patch/910453/.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- list mandatory deps before removing them
- fix flake8 warnings
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This test is a simple "Hello, World" integration test of the OpenJDK
package.
It compiles the Java app on the host, then runs it on an emulated
AARCH64 target and verifies "Hello, World" is printed.
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since version 2.29, glibc requires python 3.4 or later to build the
GNU C Library [1].
We add a new check to verify the version of python3 interpreter
installed on the host. If no suitable python3 interpreter is found,
define BR2_PYTHON3_HOST_DEPENDENCY to add host-python3 in package
dependencies when needed.
[1] https://www.sourceware.org/ml/libc-alpha/2019-01/msg00723.html
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Adam Duskett <aduskett@gmail.com>
Reviewed-by: Adam Duskett <aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: drop not so useful comment in the .mk file, as suggested by
Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This reverts commit 81771cfcdc.
The download of sha1 of a special ref currently works or not depending
on the git client version in use.
With git version 2.11.0 (present in the docker image) it does not work.
With git version 2.17.1 it works.
For the sake of reproducibility, remove this part of the TestGitRefs
test case until some code gets added to the download infra to handle
sha1 of a special ref for any git client version.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/158295269
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
gerbera package in version 1.3 unfortunately now requires CMake >= 3.8
for C++17 macros:
b5fd39f30f
So we need to bump our requirement from 3.1 to 3.8. If the host doesn't
have a CMake >= 3.8, Buildroot will build its own host-cmake package.
Also drop patch that relax cmake requirement on json-for-modern-cpp
Fixes:
- http://autobuild.buildroot.org/results/6405647b47b132ff5d0d211b92d407322d52d507
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit replaces the two RISC-V configurations used for the
autobuilders to use pre-built external toolchains rather than internal
toolchains. This saves quite a bit of build time in the autobuilders,
and also allows people to reproduce build issues in a much more
efficient way, since rebuilding the toolchain is not needed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
All toolchains have been rebuilt with Buildroot 2019.02-rc1.
Changes:
- Toolchains that were using no-longer maintained kernel headers
versions have been changed to use a variety of newer kernel headers
versions (4.4, 4.9 or 4.14).
- Since gcc 7.x is now the default in Buildroot, most toolchains that
simply use the default gcc version use 7.x instead of 6.x.
- br-arm-cortex-a9-glibc uses gcc 8.x, binutils 2.31 and kernel
headers 4.20
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
runc (which is a reverse dependency of docker-engine) is about to gain a
!uclibc dependency, so move to a glibc toolchain instead.
There are currently no prebuilt x86_64 / core2 / glibc toolchains available,
so instead use the internal toolchain backend to build one.
While we are at it, drop the infra.basetest.BASIC_TOOLCHAIN_CONFIG
reference, as that ARM toolchain configuration doesn't make any sense for
this x86-64 based test.
add docker / docker-compose tests
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following flake8 warnings:
support/scripts/pkg-stats:34:2: W605 invalid escape sequence '\$'
support/scripts/pkg-stats:34:4: W605 invalid escape sequence '\('
support/scripts/pkg-stats:34:11: W605 invalid escape sequence '\$'
support/scripts/pkg-stats:34:13: W605 invalid escape sequence '\('
support/scripts/pkg-stats:34:32: W605 invalid escape sequence '\)'
support/scripts/pkg-stats:34:34: W605 invalid escape sequence '\)'
support/scripts/pkg-stats:35:2: W605 invalid escape sequence '\s'
support/scripts/pkg-stats:35:14: W605 invalid escape sequence '\S'
support/scripts/pkg-stats:35:17: W605 invalid escape sequence '\s'
support/scripts/pkg-stats:42:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:587:133: E501 line too long (157 > 132 characters)
Note that the "invalid escape sequence" errors work because Python
leaves the \ in place if it doesn't recognise the escape sequence. But
it's better practice to use a raw string for regular expressions.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Introduce support/scripts/check-merged-usr.sh, a script that check if a
given path complies to the merged /usr requirements:
/
/bin -> usr/bin
/lib -> usr/lib
/sbin -> usr/sbin
/usr/bin/
/usr/lib/
/usr/sbin/
Use this script in skeleton-custom.mk instead of a bunch of variables
filled by $(shell ...) macros. The same script will be used to check
rootfs overlays, in a forthcoming change.
Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This commit adds fetching the latest upstream version of each package
from release-monitoring.org.
The fetching process first tries to use the package mappings of the
"Buildroot" distribution [1]. This mapping mechanism allows to tell
release-monitoring.org what is the name of a package in a given
distribution/build-system. For example, the package xutil_util-macros
in Buildroot is named xorg-util-macros on release-monitoring.org. This
mapping can be seen in the section "Mappings" of
https://release-monitoring.org/project/15037/.
If there is no mapping, then it does a regular search, and within the
search results, looks for a package whose name matches the Buildroot
name.
Even though fetching from release-monitoring.org is a bit slow, using
multiprocessing.Pool has proven to not be reliable, with some requests
ending up with an exception. So we keep a serialized approach, but
with a single HTTPSConnectionPool() for all queries. Long term, we
hope to be able to use a database dump of release-monitoring.org
instead.
From an output point of view, the latest version column:
- Is green when the version in Buildroot matches the latest upstream
version
- Is orange when the latest upstream version is unknown because the
package was not found on release-monitoring.org
- Is red when the version in Buildroot doesn't match the latest
upstream version. Note that we are not doing anything smart here:
we are just testing if the strings are equal or not.
- The cell contains the link to the project on release-monitoring.org
if found.
- The cell indicates if the match was done using a distro mapping, or
through a regular search.
[1] https://release-monitoring.org/distro/Buildroot/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Replace all YAML anchors with the new "extends" keyword because it is
more readable and more flexible (it works across configuration files
combined with the new "include" keyword).
Readability is more meaningful in .gitlab-ci.yml.in.
In the part of .gitlab-ci.yml that is auto-generated by 'make
.gitlab-ci.yml' keep the keyword in the same line of the job name.
So instead of this:
zynqmp_zcu106_defconfig:
extends: .defconfig
tests.boot.test_atf.TestATFAllwinner:
extends: .runtime_test
Use this:
zynqmp_zcu106_defconfig: { extends: .defconfig }
tests.boot.test_atf.TestATFAllwinner: { extends: .runtime_test }
Do this to to keep .gitlab-ci.yml easier to be post-processed by a
script.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a special ref to the static repo and check on the git refs test case
the download of a git package:
- with the sha1 of a special ref as version;
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: change to use the sha1 of a special ref instead of the name]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a tag to the static repo and check on the git refs test case the
download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Enables the test to use the new non-emulator base class which takes
significantly less test time.
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The git tests don't need to do a full build, they only need to do a
configure and download and/or legal-info. More tests of that type will
be added in the future. Therefore, we want to have a test base class
that doesn't automatically do a full build in the setUp().
Add this new class as a superclass of the existing BRTest class, so we
don't need to update existing tests. Only the code in run-tests that
iterates over all subclasses of BRTest has to be adapted to use
BRConfigTest instead.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add two submodules as static repos, add a branch to the main static repo
and check on the git refs test case the download of a git package:
- repo with submodule but without support in the package;
- repo with recursive submodules with support in the package.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: remove handling of inconsistent tarball hashes - that's an
actual bug that should be fixed]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Build for x86-64 as public containers in general are only available for
x86-64. Docker needs a number of kernel options enabled, so use a custom
kernel config based on the qemu one.
Docker needs entropy at startup, so enable the virtio-rng-pci device to
expose entropy to the guest. The default RAM amount (128M) is not enough to
run docker / docker-compose, so bump to 512MB.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The scp download helper is broken when the server URL starts with 'scp://'.
Such prefix is used in two situations:
1. to let FOO_SITE point to an scp location without explicitly having to set
'FOO_SITE_METHOD = scp'
2. when BR2_PRIMARY_SITE or BR2_BACKUP_SITE points to an scp location. In
this case, there is no equivalent of 'SITE_METHOD'.
Strip out the scheme prefix, similarly to how the 'file' download helper
does it. That helper has the same cases as above.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The password is used in multiple places, so add a constant for it instead of
hardcoding it multiple times.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a branch to the static repo and check on the git refs test case the
download of a git package:
- with a sha1 reachable by a branch name, but not pointed by it, as
version. This is the most common use case for git refs in the tree;
- with a partial sha1 of a commit reachable by a branch as version;
- with a sha1 of the commit head of a branch as version;
- with a partial sha1 of the commit head of a branch as version;
Enforce the download always occurs by removing the BR2_DL_DIR used for
the tarballs generated by the git download infra.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
setlocalversion will use 'hg id' to determine whether or not the current
revision is tagged. If there is no tag, the Mercurial revision is printed,
otherwise nothing is printed.
The problem is that the user may have custom configuration settings (in
their ~/.hgrc file or similar) that changes the output of 'hg id' in a way
that the script does not expect. In such cases, the Mercurial revision may
not be printed or printed incorrectly.
It is good practice to ignore the user environment when calling Mercurial
commands from a well-defined script, by setting the environment variable
HGRCPATH to the empty string. See also 'hg help environment'.
In the particular case of Nokia, a custom extension adds dynamic tags in the
repository, i.e. tags that are stored in a file external to the repository
and only visible when the extension is active. These tags should not
influence the behavior of setlocalversion as they are not official Buildroot
tags, i.e. even if a revision is tagged, the Mercurial revision should still
be printed.
Note that this still does not solve the problem where an organization adds
_real_ tags in their Buildroot repository. For example, there might be a
moving tag 'last-validated' or tags indicating in which product release that
Buildroot revision was used. In these cases, setlocalversion will still not
behave as expected, i.e. show the Mercurial revision.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When Buildroot is stored in a Mercurial repository on a branch other than
'default' ('master' in git terms), setlocalversion (used to populate
/etc/os-release) will incorrectly think that this is a tagged version and
will NOT print out the revision hash.
This is due to the fact that the output of 'hg id' is assumed to be
"<revision> <tags-if-any>"
but when on a branch it actually is:
"<revision> (<branch>) <tags-if-any>"
To let setlocalversion receive the output it expects, explicitly ask 'hg id'
to retrieve only the revision hash and any tags, ommitting any branch
information.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
All upcoming tests for git refs will rely on the return code of make to
determine whether a git ref can be downloaded or not and also to
determine whether the downloaded content is correct (all of this taking
advantage of the check-hash mechanism already in place for git
packages).
So to avoid false results i.e. in the case the check-hash mechanism
become broken in the master branch, add some sanity checks before the
actual test of download git refs.
Add the minimum test case for git refs containing only sanity checks.
Reuse the commit in the static repo.
Add a br2-external with two packages to check that:
- trying to download an invalid sha1 generates an error;
- downloading a valid sha1 that contains unexpected content generates
an error.
In order to ease the maintenance and review, each upcoming patch adding
checks to this test case will add at same time the commits to the static
repo, the equivalent packages to the br2-external and code to the test
case.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add one test case to ensure the hash is checked for git packages:
- correct hash;
- wrong hash;
- no hash file.
Add required infra:
- a GitRemote class, that can start a git server in the host machine to
emulate a remote git server under the control of the test;
- a new base class, called GitTestBase, that inherits from BRTest and
must be subclassed by all git test cases.
Its setUp() method takes care of configuring the build with a
br2-external, avoiding to hit http://sources.buildroot.net by using
an empty BR2_BACKUP_SITE. It also avoids downloading not
pre-installed dependencies (i.e. lzip) every time by calling 'make
dependencies' using the common dl directory, and it instantiates the
GitRemote object.
Besides the Python scripts, add some fixtures used during the tests:
- a br2-external (git-hash) with one package for each part of the test
case;
- a static git bare repo (repo.git) to be served using GitRemote class.
Neither the br2-external nor the check hash functionalities are the
subject of these tests per se, so for simplicity limit the check to the
error codes and don't look for the messages in the log.
Thanks to Arnout for the hint about how to add a bare repo to test.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Arnout: split long line; reorder imports to satisfy flake8]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
