kumquat-buildroot/support/scripts/generate-gitlab-ci-yml

144 lines
4.1 KiB
Plaintext
Raw Normal View History

Makefile: offload .gitlab-ci.yml generation GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: | only: | - /-$CI_JOB_NAME$/ | - /-\$CI_JOB_NAME$/ | - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: | only: | refs: | - branches | - tags | variables: | - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: |.runtime_test: &runtime_test | only: | - tags |tests.package.test_python_txaio.TestPythonPy2Txaio: | <<: *runtime_test | only: | - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-29 00:58:38 +01:00
#!/usr/bin/env bash
set -e
set -o pipefail
main() {
local template="${1}"
Makefile: offload .gitlab-ci.yml generation GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: | only: | - /-$CI_JOB_NAME$/ | - /-\$CI_JOB_NAME$/ | - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: | only: | refs: | - branches | - tags | variables: | - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: |.runtime_test: &runtime_test | only: | - tags |tests.package.test_python_txaio.TestPythonPy2Txaio: | <<: *runtime_test | only: | - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-29 00:58:38 +01:00
preamble "${template}"
gen_tests
}
preamble() {
local template="${1}"
cat - "${template}" <<-_EOF_
# This file is generated; do not edit!
# Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
image: ${CI_JOB_IMAGE}
_EOF_
}
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
gen_tests() {
local -a basics defconfigs runtimes
utils/test-pkg: add gitlab-ci support The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing. To trigger this, a developer will have to add, in the latest commit of their branch, a token on its own line, followed by a configuration fragment, e.g.: test-pkg config: SOME_OPTION=y # OTHER_OPTION is not set SOME_VARIABLE="some value" This configuration fragment is used as input to test-pkg. To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per test) in the child pipeline. We use the newly-introduced --prepare-only mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "needs::pipeline" and "needs::job" directives. This is a bit tricky, and is best described by the Gitlab-CI documentation [0]. We also list those .config files to generate the actual list of jobs to run in the child pipeline. Notes: - if the user provides an empty fragment, this is considered an error: indeed, without a fragment (and the package name), there is no way to know what to test; - if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error. [0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [yann.morin.1998@free.fr: - split the change to test-pkg to its own patch - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg - some code-style-candies... ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
local do_basics do_defconfigs do_runtime do_testpkg
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
local defconfigs_ext cfg tst
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
basics=( DEVELOPERS flake8 package )
defconfigs=( $(cd configs; LC_ALL=C ls -1 *_defconfig) )
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
runtimes=( $(./support/testing/run-tests -l 2>&1 \
| sed -r -e '/^test_run \((.*)\).*/!d; s//\1/' \
| LC_ALL=C sort)
)
if [ -n "${CI_COMMIT_TAG}" ]; then
support/scripts: don't build board defconfigs with Gitlab's pipelines trigged on tag Currently when a tag is added to the Buildroot git tree, the gitlab-ci create a pipeline with several hundred of jobs (~750) to build all defconfigs and execute the Buildroot testsuite. However, there is only a limited number of gitlab-ci runner (9 runners) and some jobs reach the timeout limit (24h) while waiting for a runner [1]. Indeed, the Buildroot project doesn't use the Gitlab's shared runners. In addition to the pipeline created when a new tag is added to the git repository, two pipelines are created each weeks to execute the Buildroot testsuite (on monday [2]) and build all defconfigs (on Thursday [3]). At some point there are too many jobs waiting in gitlab due board defconfigs builds. Indded a board defconfig requires a lot of time (~30min) compared to other jobs in order to build a toolchain and a kernel linux along with a basic rootfs. There is currently 262 defconfigs. This is even worse when several pipelines are trigged at the same time (new git tag and scheduled pipeline trigger). In order to reduce the number of long jobs, don't build board defconfigs with pipelines trigged on tag, keeping only the runtime tests and the Qemu's defconfigs. [1] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966541 [2] https://gitlab.com/buildroot.org/buildroot/-/pipelines/404035190 [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/401685550 Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Acked-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-11-09 23:03:28 +01:00
# When a tag is added to the Buildroot git tree, we want
# to run the runtime tests and only test Qemu defconfigs.
defconfigs=( $(cd configs; LC_ALL=C ls -1 qemu_*_defconfig) )
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
do_basics=true
do_defconfigs=base
do_runtime=true
elif [ "${CI_PIPELINE_SOURCE}" = "trigger" ]; then
case "${BR_SCHEDULE_JOBS}" in
(basic)
do_basics=true
do_defconfigs=check
defconfigs_ext=_check
;;
(defconfig)
do_defconfigs=base
;;
(runtime)
do_runtime=true
;;
esac
else
case "${CI_COMMIT_REF_NAME}" in
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
(*-basics)
do_basics=true
do_defconfigs=check
defconfigs_ext=_check
;;
(*-defconfigs)
do_defconfigs=base
;;
(*-defconfigs-*)
pattern=$(echo ${CI_COMMIT_REF_NAME} | sed 's%[^\-]*-defconfigs-\(.*\)%\1%')
defconfigs=( $(cd configs; LC_ALL=C ls -1 | grep ^${pattern}) )
do_defconfigs=base
;;
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
(*-*_defconfig)
defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
do_defconfigs=base
;;
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
(*-runtime-tests)
do_runtime=true
;;
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
(*-tests.*)
runtimes=( $(./support/testing/run-tests -l 2>&1 \
| sed -r -e '/^test_run \((.*)\).*/!d; s//\1/' \
| LC_ALL=C sort \
| grep "^${CI_COMMIT_REF_NAME##*-}")
)
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
do_runtime=true
;;
esac
fi
utils/test-pkg: add gitlab-ci support The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing. To trigger this, a developer will have to add, in the latest commit of their branch, a token on its own line, followed by a configuration fragment, e.g.: test-pkg config: SOME_OPTION=y # OTHER_OPTION is not set SOME_VARIABLE="some value" This configuration fragment is used as input to test-pkg. To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per test) in the child pipeline. We use the newly-introduced --prepare-only mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "needs::pipeline" and "needs::job" directives. This is a bit tricky, and is best described by the Gitlab-CI documentation [0]. We also list those .config files to generate the actual list of jobs to run in the child pipeline. Notes: - if the user provides an empty fragment, this is considered an error: indeed, without a fragment (and the package name), there is no way to know what to test; - if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error. [0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [yann.morin.1998@free.fr: - split the change to test-pkg to its own patch - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg - some code-style-candies... ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
# Retrieve defconfig for test-pkg from the git commit message (if any)
if grep -q -E '^test-pkg config:$' <<<"${CI_COMMIT_DESCRIPTION}"; then
sed -r -n -e '/^test-pkg config:$/{:a;n;p;ba;}' \
<<<"${CI_COMMIT_DESCRIPTION}" \
>defconfig.frag
if [ ! -s defconfig.frag ]; then
printf "Empty configuration fragment.\n" >&2; exit 1
fi
# Use --all since we expect the user having already pre-tested the
# new package with the default subset of toolchains.
./utils/test-pkg \
--all --prepare-only \
--config-snippet defconfig.frag \
--build-dir br-test-pkg >&2
do_testpkg=( $(ls -1 br-test-pkg/*/.config 2>/dev/null |xargs -r dirname ) )
if [ "${#do_testpkg[@]}" -eq 0 ]; then
printf "Configuration fragment enables no test.\n" >&2; exit 1
fi
fi
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
# If nothing else, at least do the basics to generate a valid pipeline
if [ -z "${do_defconfigs}" \
-a -z "${do_runtime}" \
utils/test-pkg: add gitlab-ci support The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing. To trigger this, a developer will have to add, in the latest commit of their branch, a token on its own line, followed by a configuration fragment, e.g.: test-pkg config: SOME_OPTION=y # OTHER_OPTION is not set SOME_VARIABLE="some value" This configuration fragment is used as input to test-pkg. To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per test) in the child pipeline. We use the newly-introduced --prepare-only mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "needs::pipeline" and "needs::job" directives. This is a bit tricky, and is best described by the Gitlab-CI documentation [0]. We also list those .config files to generate the actual list of jobs to run in the child pipeline. Notes: - if the user provides an empty fragment, this is considered an error: indeed, without a fragment (and the package name), there is no way to know what to test; - if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error. [0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [yann.morin.1998@free.fr: - split the change to test-pkg to its own patch - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg - some code-style-candies... ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
-a -z "${do_testpkg}" \
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
]
then
do_basics=true
fi
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
if ${do_basics:-false}; then
for tst in "${basics[@]}"; do
printf 'check-%s: { extends: .check-%s_base }\n' "${tst}" "${tst}"
done
fi
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
if [ -n "${do_defconfigs}" ]; then
for cfg in "${defconfigs[@]}"; do
printf '%s%s: { extends: .defconfig_%s }\n' \
"${cfg}" "${defconfigs_ext}" "${do_defconfigs}"
done
fi
support/scripts/generate-gitlab-ci-yml: rework generation of pipelines Currently, we handle three kinds of tests: basic, defconfig, and runtime, and we treat them totally independently ones from the others. Except for the basic tests that are ignored when defconfig or runtime tests are explicitly requested. The basic tests are also run systematically on all our reference branches: master, next (when it exists), and the maintenance branches: YYYY.MM.x. Furthermore, we can see that the conditions to run each set of tests are very similar, with only the explicit queries differing by name. Rework the script so that the conditions are expressed only once, and each set of tests is decided for each condition. This makes it easier to decide what tests should run under what conditions. Using GitLab-CI's schedules, with a variable expressing the actual test to run, would seem the obvious choice to trigger the pipelines. However, a schedule is configured for a specific branch, which means we would need one schedule per branch we want to build per test cases we want to run, *and* that we update those schedules when we add/remove branches (e.g. when we open/close 'next', or a maintenance branch). This is not very nice, as it requires some manual tweaking and twiddling on the web UI. Instead, we resort to using triggers, that will be triggered from a cronjob on some server. Using a cronjiob allows us to more easily manage the branches we want to test and test cases we want to run, to more easily spread the load over the week, etc... Note: triggering a pipeline can be done with a simple curl invocation: $ curl -X POST \ -F "token=${YOUR_TOKEN}" \ -F "ref=${BRANCH_TO_TEST}" \ -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \ "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline" Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-05 22:48:25 +02:00
if ${do_runtime:-false}; then
printf '%s: { extends: .runtime_test_base }\n' "${runtimes[@]}"
fi
utils/test-pkg: add gitlab-ci support The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing. To trigger this, a developer will have to add, in the latest commit of their branch, a token on its own line, followed by a configuration fragment, e.g.: test-pkg config: SOME_OPTION=y # OTHER_OPTION is not set SOME_VARIABLE="some value" This configuration fragment is used as input to test-pkg. To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per test) in the child pipeline. We use the newly-introduced --prepare-only mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "needs::pipeline" and "needs::job" directives. This is a bit tricky, and is best described by the Gitlab-CI documentation [0]. We also list those .config files to generate the actual list of jobs to run in the child pipeline. Notes: - if the user provides an empty fragment, this is considered an error: indeed, without a fragment (and the package name), there is no way to know what to test; - if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error. [0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [yann.morin.1998@free.fr: - split the change to test-pkg to its own patch - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg - some code-style-candies... ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-21 15:46:46 +02:00
if [ -n "${do_testpkg}" ]; then
printf '%s: { extends: .test_pkg }\n' "${do_testpkg[@]}"
fi
}
main "${@}"