Notes: We can't use runtime_test_download job from the parent pipeline
(generate-gitlab-ci) since the artifacts archive size is limited to 5MB.
So introduce a new custom stage named "download" executed before "test"
stage. test-dl directory that contain downloaded files can be an
artifact of the job passed to all jobs of next stages.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/4409032417
Runtime tested:
https://gitlab.com/kubu93/buildroot/-/pipelines/934319226
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
"stages" is curently not defined in the .gitlab-ci.yml file,
so the default pipeline stages are:
.pre
build
test
deploy
.post
Since any jobs specify a stage, all jobs are assigned the "test"
stage [1]. All other stages defined by default are not used in the
gitlab-ci pipeline, they remain hidden.
In order to introduce a new custom stage, add the "test" stage
explicitely.
[1] https://docs.gitlab.com/ee/ci/yaml/#stages
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 7814dbce15)
[Peter: drop Makefile update]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 8025cfad10)
[Peter: drop Makefile hunk]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)
Expand the runtime test and the unit tests for check-package.
Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 71ddf1a084)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation.
It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.
The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).
Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).
[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .
The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).
Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.
At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
(for legacy handling) but is referenced in the package without a
comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
but it is not actually referenced.
There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.
Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'
At same time the script is created:
- add unit tests for it, they can be run using:
utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
using the script) and check-check-symbols (to check the script against
its unit tests)
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit d48a8beb39)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 25680e6aa8)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 25d865996d)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit bf0d8c9659)
[Peter: drop Makefile changes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
python-scipy is moving to a meson build infrastructure, and requires
fortran cross-compilation. This commit extends the cross-compilation
configuration file generated for use by meson to add the "fortran",
"fortran_args" and "fortran_link_args" properties.
Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 2e399893b4)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When compiling Qt6 applications for the target, CMake needs to have
two variables defined to find the host installation of Qt. These two
variables are unconditionally defined, regardless of whether Qt6 is
enabled in the configuration or not, as they do no harm when Qt6 is
not present/used.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Now that we have a working mechanism to validate the DEVELOPERS file
with the get-developers -v option, let's use it.
This brings back proper validation of the DEVELOPERS file, as
get-developers without argument no longer did any validation following
commit 45aabcddc5 utils/get-developers: really make it callable from
elsewhere than the toplevel directory".
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some packages such as libclc need to override cmake toolchain
variables, to avoid errors caused by trying to set overriden
variables ensure that they are not defined before being set.
This prevents difficult to debug silent dropping of overriden
variables.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
... so we can catch regressions on check-package.
Update to the new docker image that was pushed after the previous
commit.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Removed a few variables, as they were only used to communicate
between the meson package and pkg-meson.mk and are not needed
anymore.
Moved cross-compilation.conf.in out of meson package.
Creating the cross-compilation.conf files for packages is now
using the original template.
To avoid duplicate code, the common sed pattern is stored in
a make variable.
Use explicit Buildroot variables for compiler tools,
and some fixes. (TARGET_LDFLAGS and TARGET_CXXFLAGS
were mixed up with PKG_TARGET_CFLAGS)
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Arnout: keep PKG_MESON_INSTALL_CROSS_CONF in
TOOLCHAIN_TARGET_FINALIZE_HOOKS]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 323ae1e681)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, relocate-sdk.sh must be run _after_ relocating the SDK. There
are cases where it is useful to already prepare the SDK _before_
relocating. For example, it allows to prepare a tarball that the user
has to extract to a specific, pre-defined location and nothing more than
that, which is simpler for the user than requiring the script to be run.
In addition, it hides the build directory that was used by the SDK
builder (somewhat).
Add an optional argument to relocate-sdk.sh that gives the target
directory.
Signed-off-by: Gleb Mazovetskiy <glex.spb@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, we have two functions that build a comma-separated list
of items; one is double-quoting the items, while the other is
single-quoting them. Their naming is not very consistent.
Besides, in a followup change, we will need to build a comma-separated
list of items that are already double-quoted.
Introduce a macro that does just build a comma-separated list, and
use that in the two other macros; rename the existing macro so the
naming is consistent.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit d815599e37)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 516b837002)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The gitlab-ci support in test-pkg allows to parallelize the test-pkg
work into several gitlab jobs. It's much faster than local serialized
testing.
To trigger this, a developer will have to add, in the latest commit of
their branch, a token on its own line, followed by a configuration
fragment, e.g.:
test-pkg config:
SOME_OPTION=y
# OTHER_OPTION is not set
SOME_VARIABLE="some value"
This configuration fragment is used as input to test-pkg.
To be able to generate one job per test to run, we need the list of
tests in the parent pipeline, and the individual .config files (one per
test) in the child pipeline. We use the newly-introduced --prepare-only
mode to test-pkg, and collect all the generated .config files as
artefacts; those are inherited in the child pipeline via the
"needs::pipeline" and "needs::job" directives. This is a bit tricky,
and is best described by the Gitlab-CI documentation [0].
We also list those .config files to generate the actual list of jobs to
run in the child pipeline.
Notes:
- if the user provides an empty fragment, this is considered an error:
indeed, without a fragment (and the package name), there is no way
to know what to test;
- if that fragment yields an empty list of tests, then there is
nothing to test either, so that is also considered an error.
[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998@free.fr:
- split the change to test-pkg to its own patch
- generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
listing the .config files created by test-pkg
- some code-style-candies...
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit d5127a4de7)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In a followup commit, the make command used to log and display the last
lines on error will be used in another job.
Factorize it by introducing .run_make template.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Parallelizes locale generation based on `BR2_JLEVEL` setting.
Locale generation always runs during the finalize stage and can consume
a significant amount of time. Parallelizing it greatly reduces that time
on multi-core machines.
To parallelize it, we first invoke `localedef` for every locale in
parallel with the `--no-archive` option. This creates the intermediate
locale data instead of writing to the finally archive directly.
Then, we invoke `localedef` again once to create the archive from the
intermediate compiled locale data files.
We have to do it this way because `localedef` does not do any locking
when writing to the archive file, so calling it without `--no-archive`
concurrently could result in a corrupt archive file or an archive file
that is missing some locales.
While we're at it, make two additional improvements:
- Remove locale-archive before adding to it. Otherwise, repeated
applications of target-finalize will keep on growing the file.
- Sort the locales when creating locale-archive so its contents are
reproducible.
We use `find` to collect the installed locales rather than LOCALES. This
makes it possible for something else (skeleton, overlay, custom package)
to create and install additional locales and still have them added to
locale-archive.
Signed-off-by: Gleb Mazovetskiy <glex.spb@gmail.com>
[Arnout:
- Remove -j$(PARALLEL_JOBS), it's already part of $(MAKE)
- Remove HOST_DIR, TARGET_DIR, STAGING_DIR, they're already exported
- Extend commit message
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 76b4f9e9b6)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit bb10b0dfe6)
[Peter: drop Makefile change]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
