gitlab-ci: fix pipelines
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation.
It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.
The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).
Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).
[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:33:12 +01:00
|
|
|
before_script:
|
|
|
|
|
- git config --global --add safe.directory ${CI_PROJECT_DIR}
|
|
|
|
|
|
2022-01-23 17:08:45 +01:00
|
|
|
.check-check-package_base:
|
|
|
|
|
script:
|
|
|
|
|
- python3 -m pytest -v utils/checkpackagelib/
|
|
|
|
|
|
2022-11-27 14:07:39 +01:00
|
|
|
.check-check-symbol_base:
|
|
|
|
|
script:
|
|
|
|
|
- python3 -m pytest -v utils/checksymbolslib/
|
|
|
|
|
|
2020-09-06 22:12:27 +02:00
|
|
|
.check-DEVELOPERS_base:
|
2017-06-30 18:42:53 +02:00
|
|
|
script:
|
2022-07-23 16:07:18 +02:00
|
|
|
- utils/get-developers -v
|
2017-06-30 18:42:53 +02:00
|
|
|
|
2020-09-06 22:12:27 +02:00
|
|
|
.check-package_base:
|
2017-11-30 00:08:45 +01:00
|
|
|
script:
|
2018-08-11 12:44:23 +02:00
|
|
|
- make check-package
|
2017-11-30 00:08:45 +01:00
|
|
|
|
2022-11-27 14:07:39 +01:00
|
|
|
.check-symbol_base:
|
|
|
|
|
script:
|
|
|
|
|
- utils/check-symbols
|
|
|
|
|
|
2020-07-27 17:51:27 +02:00
|
|
|
.defconfig_check:
|
|
|
|
|
script:
|
gitlab-ci: don't use before_script in job templates
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:33:11 +01:00
|
|
|
- DEFCONFIG_NAME=$(echo ${CI_JOB_NAME} | sed -e 's,_check$,,g')
|
2020-07-27 17:51:27 +02:00
|
|
|
- echo "Configure Buildroot for ${DEFCONFIG_NAME}"
|
|
|
|
|
- make ${DEFCONFIG_NAME}
|
|
|
|
|
- support/scripts/check-dotconfig.py .config configs/${DEFCONFIG_NAME}
|
|
|
|
|
artifacts:
|
|
|
|
|
when: on_failure
|
|
|
|
|
expire_in: 2 weeks
|
|
|
|
|
paths:
|
|
|
|
|
- .config
|
|
|
|
|
|
2021-06-28 22:15:12 +02:00
|
|
|
.run_make: &run_make
|
|
|
|
|
|
|
|
|
|
|
make O=${OUTPUT_DIR} > >(tee build.log |grep '>>>') 2>&1 || {
|
|
|
|
|
echo 'Failed build last output'
|
|
|
|
|
tail -200 build.log
|
|
|
|
|
exit 1
|
|
|
|
|
}
|
|
|
|
|
|
2019-04-08 05:22:54 +02:00
|
|
|
.defconfig_base:
|
gitlab-ci: don't use before_script in job templates
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:33:11 +01:00
|
|
|
script:
|
2020-09-06 22:12:28 +02:00
|
|
|
- DEFCONFIG_NAME=${CI_JOB_NAME}
|
2021-06-28 22:15:12 +02:00
|
|
|
- OUTPUT_DIR=output
|
2019-04-08 05:22:54 +02:00
|
|
|
- echo "Configure Buildroot for ${DEFCONFIG_NAME}"
|
|
|
|
|
- make ${DEFCONFIG_NAME}
|
2020-07-27 17:51:26 +02:00
|
|
|
- ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
|
2019-04-08 05:22:53 +02:00
|
|
|
- echo 'Build buildroot'
|
2021-06-28 22:15:12 +02:00
|
|
|
- *run_make
|
2020-02-17 21:50:30 +01:00
|
|
|
- |
|
|
|
|
|
./support/scripts/boot-qemu-image.py "${DEFCONFIG_NAME}" > >(tee runtime-test.log) 2>&1 || {
|
|
|
|
|
echo 'Failed runtime test last output'
|
|
|
|
|
tail -200 runtime-test.log
|
|
|
|
|
exit 1
|
|
|
|
|
}
|
2017-02-14 00:23:03 +01:00
|
|
|
artifacts:
|
|
|
|
|
when: always
|
|
|
|
|
expire_in: 2 weeks
|
|
|
|
|
paths:
|
2018-12-08 18:13:42 +01:00
|
|
|
- .config
|
2017-02-14 00:23:03 +01:00
|
|
|
- build.log
|
|
|
|
|
- output/images/
|
|
|
|
|
- output/build/build-time.log
|
|
|
|
|
- output/build/packages-file-list.txt
|
2018-12-08 18:13:42 +01:00
|
|
|
- output/build/*/.config
|
2020-02-17 21:50:30 +01:00
|
|
|
- runtime-test.log
|
2019-04-08 05:22:54 +02:00
|
|
|
|
|
|
|
|
.runtime_test_base:
|
2017-07-30 06:49:43 +02:00
|
|
|
# Keep build directories so the rootfs can be an artifact of the job. The
|
|
|
|
|
# runner will clean up those files for us.
|
2017-08-05 04:05:20 +02:00
|
|
|
# Multiply every emulator timeout by 10 to avoid sporadic failures in
|
|
|
|
|
# elastic runners.
|
2019-04-08 05:22:54 +02:00
|
|
|
script:
|
gitlab-ci: don't use before_script in job templates
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:33:11 +01:00
|
|
|
- TEST_CASE_NAME=${CI_JOB_NAME}
|
2019-04-08 05:22:54 +02:00
|
|
|
- echo "Starting runtime test ${TEST_CASE_NAME}"
|
|
|
|
|
- ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${TEST_CASE_NAME}
|
2017-07-02 18:13:22 +02:00
|
|
|
artifacts:
|
|
|
|
|
when: always
|
|
|
|
|
expire_in: 2 weeks
|
|
|
|
|
paths:
|
|
|
|
|
- test-output/*.log
|
2017-12-20 15:19:33 +01:00
|
|
|
- test-output/*/.config
|
2017-07-30 06:49:43 +02:00
|
|
|
- test-output/*/images/*
|
2019-04-08 05:22:54 +02:00
|
|
|
|
utils/test-pkg: add gitlab-ci support
The gitlab-ci support in test-pkg allows to parallelize the test-pkg
work into several gitlab jobs. It's much faster than local serialized
testing.
To trigger this, a developer will have to add, in the latest commit of
their branch, a token on its own line, followed by a configuration
fragment, e.g.:
test-pkg config:
SOME_OPTION=y
# OTHER_OPTION is not set
SOME_VARIABLE="some value"
This configuration fragment is used as input to test-pkg.
To be able to generate one job per test to run, we need the list of
tests in the parent pipeline, and the individual .config files (one per
test) in the child pipeline. We use the newly-introduced --prepare-only
mode to test-pkg, and collect all the generated .config files as
artefacts; those are inherited in the child pipeline via the
"needs::pipeline" and "needs::job" directives. This is a bit tricky,
and is best described by the Gitlab-CI documentation [0].
We also list those .config files to generate the actual list of jobs to
run in the child pipeline.
Notes:
- if the user provides an empty fragment, this is considered an error:
indeed, without a fragment (and the package name), there is no way
to know what to test;
- if that fragment yields an empty list of tests, then there is
nothing to test either, so that is also considered an error.
[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998@free.fr:
- split the change to test-pkg to its own patch
- generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
listing the .config files created by test-pkg
- some code-style-candies...
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
|
|
|
.test_pkg:
|
|
|
|
|
stage: build
|
|
|
|
|
script:
|
gitlab-ci: don't use before_script in job templates
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-07 22:33:11 +01:00
|
|
|
- OUTPUT_DIR=${CI_JOB_NAME}
|
utils/test-pkg: add gitlab-ci support
The gitlab-ci support in test-pkg allows to parallelize the test-pkg
work into several gitlab jobs. It's much faster than local serialized
testing.
To trigger this, a developer will have to add, in the latest commit of
their branch, a token on its own line, followed by a configuration
fragment, e.g.:
test-pkg config:
SOME_OPTION=y
# OTHER_OPTION is not set
SOME_VARIABLE="some value"
This configuration fragment is used as input to test-pkg.
To be able to generate one job per test to run, we need the list of
tests in the parent pipeline, and the individual .config files (one per
test) in the child pipeline. We use the newly-introduced --prepare-only
mode to test-pkg, and collect all the generated .config files as
artefacts; those are inherited in the child pipeline via the
"needs::pipeline" and "needs::job" directives. This is a bit tricky,
and is best described by the Gitlab-CI documentation [0].
We also list those .config files to generate the actual list of jobs to
run in the child pipeline.
Notes:
- if the user provides an empty fragment, this is considered an error:
indeed, without a fragment (and the package name), there is no way
to know what to test;
- if that fragment yields an empty list of tests, then there is
nothing to test either, so that is also considered an error.
[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998@free.fr:
- split the change to test-pkg to its own patch
- generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
listing the .config files created by test-pkg
- some code-style-candies...
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
|
|
|
- echo "Configure Buildroot for ${OUTPUT_DIR}"
|
|
|
|
|
- make O=${OUTPUT_DIR} syncconfig
|
|
|
|
|
- make O=${OUTPUT_DIR} savedefconfig
|
|
|
|
|
- echo 'Build buildroot'
|
|
|
|
|
- *run_make
|
|
|
|
|
needs:
|
|
|
|
|
- pipeline: $PARENT_PIPELINE_ID
|
|
|
|
|
job: generate-gitlab-ci-yml
|
|
|
|
|
artifacts:
|
|
|
|
|
when: always
|
|
|
|
|
expire_in: 2 weeks
|
|
|
|
|
paths:
|
|
|
|
|
- build.log
|
|
|
|
|
- br-test-pkg/*/.config
|
|
|
|
|
- br-test-pkg/*/defconfig
|
|
|
|
|
- br-test-pkg/*/build/build-time.log
|
|
|
|
|
- br-test-pkg/*/build/packages-file-list*.txt
|
