Makefile: offload .gitlab-ci.yml generation
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
| only:
| - /-$CI_JOB_NAME$/
| - /-\$CI_JOB_NAME$/
| - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
| only:
| refs:
| - branches
| - tags
| variables:
| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
| only:
| - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
| <<: *runtime_test
| only:
| - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.
So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.
Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.
Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.
While moving the commands to the script:
- mimic the behavior of the previous make target and fail on any
command that fails, by using 'set -e';
- break the original lines in one command per line, making the diff for
any patch to be applied to this file to look nicer;
- keep the script as simple as possible, without functions, just a
script that executes from the top to bottom;
- do not perform validations on the input parameters, any command that
fails already makes the script to fail;
- do not add an usage message, the script is not intended to be called
directly.
This patch does not change functionality.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-29 00:58:38 +01:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
set -o pipefail
|
|
|
|
|
2020-09-06 22:12:20 +02:00
|
|
|
main() {
|
|
|
|
local template="${1}"
|
Makefile: offload .gitlab-ci.yml generation
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
| only:
| - /-$CI_JOB_NAME$/
| - /-\$CI_JOB_NAME$/
| - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
| only:
| refs:
| - branches
| - tags
| variables:
| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
| only:
| - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
| <<: *runtime_test
| only:
| - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.
So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.
Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.
Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.
While moving the commands to the script:
- mimic the behavior of the previous make target and fail on any
command that fails, by using 'set -e';
- break the original lines in one command per line, making the diff for
any patch to be applied to this file to look nicer;
- keep the script as simple as possible, without functions, just a
script that executes from the top to bottom;
- do not perform validations on the input parameters, any command that
fails already makes the script to fail;
- do not add an usage message, the script is not intended to be called
directly.
This patch does not change functionality.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-29 00:58:38 +01:00
|
|
|
|
2020-09-06 22:12:21 +02:00
|
|
|
preamble "${template}"
|
Makefile: offload .gitlab-ci.yml generation
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
| only:
| - /-$CI_JOB_NAME$/
| - /-\$CI_JOB_NAME$/
| - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
| only:
| refs:
| - branches
| - tags
| variables:
| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
| only:
| - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
| <<: *runtime_test
| only:
| - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.
So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.
Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.
Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.
While moving the commands to the script:
- mimic the behavior of the previous make target and fail on any
command that fails, by using 'set -e';
- break the original lines in one command per line, making the diff for
any patch to be applied to this file to look nicer;
- keep the script as simple as possible, without functions, just a
script that executes from the top to bottom;
- do not perform validations on the input parameters, any command that
fails already makes the script to fail;
- do not add an usage message, the script is not intended to be called
directly.
This patch does not change functionality.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-29 00:58:38 +01:00
|
|
|
|
2020-09-06 22:12:27 +02:00
|
|
|
gen_basics
|
2020-09-06 22:12:22 +02:00
|
|
|
gen_defconfigs
|
2020-09-06 22:12:24 +02:00
|
|
|
gen_tests
|
2020-09-06 22:12:20 +02:00
|
|
|
}
|
|
|
|
|
2020-09-06 22:12:21 +02:00
|
|
|
preamble() {
|
|
|
|
local template="${1}"
|
|
|
|
|
|
|
|
cat - "${template}" <<-_EOF_
|
|
|
|
# This file is generated; do not edit!
|
|
|
|
# Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
|
|
|
|
|
|
|
|
image: ${CI_JOB_IMAGE}
|
|
|
|
|
|
|
|
_EOF_
|
|
|
|
}
|
|
|
|
|
2020-09-06 22:12:27 +02:00
|
|
|
gen_basics() {
|
|
|
|
local tst
|
|
|
|
|
|
|
|
# Skip basic tests when explicitly building defconfigs or runtime tests
|
|
|
|
case "${CI_COMMIT_REF_NAME}" in
|
|
|
|
(*-defconfigs) return;;
|
|
|
|
(*-*_defconfig) return;;
|
|
|
|
(*-runtime-tests) return;;
|
|
|
|
(*-tests.*) return;;
|
|
|
|
esac
|
|
|
|
|
|
|
|
for tst in DEVELOPERS flake8 package; do
|
|
|
|
printf 'check-%s: { extends: .check-%s_base }\n' "${tst}" "${tst}"
|
|
|
|
done
|
|
|
|
}
|
|
|
|
|
2020-09-06 22:12:22 +02:00
|
|
|
gen_defconfigs() {
|
|
|
|
local -a defconfigs
|
2020-09-06 22:12:29 +02:00
|
|
|
local template cfg ext
|
2020-09-06 22:12:22 +02:00
|
|
|
|
|
|
|
defconfigs=( $(cd configs; LC_ALL=C ls -1 *_defconfig) )
|
|
|
|
|
|
|
|
if [ -n "${CI_COMMIT_TAG}" ]; then
|
|
|
|
# For tags, create a pipeline.
|
2020-09-06 22:12:29 +02:00
|
|
|
template=base
|
2020-09-06 22:12:22 +02:00
|
|
|
fi
|
|
|
|
if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
|
|
|
|
# For pipeline created by using a trigger token.
|
2020-09-06 22:12:29 +02:00
|
|
|
template=base
|
2020-09-06 22:12:22 +02:00
|
|
|
fi
|
|
|
|
case "${CI_COMMIT_REF_NAME}" in
|
2020-09-06 22:12:29 +02:00
|
|
|
# For master, next, and maintenance branches, only check the defconfigs
|
|
|
|
(master|next|????.??.x)
|
|
|
|
template=check
|
|
|
|
ext=_check
|
|
|
|
;;
|
2020-09-06 22:12:22 +02:00
|
|
|
# For the branch or tag name named *-defconfigs, create a pipeline.
|
2020-09-06 22:12:25 +02:00
|
|
|
(*-defconfigs)
|
2020-09-06 22:12:29 +02:00
|
|
|
template=base
|
2020-09-06 22:12:25 +02:00
|
|
|
;;
|
|
|
|
(*-*_defconfig)
|
|
|
|
defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
|
2020-09-06 22:12:29 +02:00
|
|
|
template=base
|
2020-09-06 22:12:25 +02:00
|
|
|
;;
|
2020-09-06 22:12:22 +02:00
|
|
|
esac
|
|
|
|
|
2020-09-06 22:12:29 +02:00
|
|
|
if [ -n "${template}" ]; then
|
|
|
|
for cfg in "${defconfigs[@]}"; do
|
|
|
|
printf '%s%s: { extends: .defconfig_%s }\n' \
|
|
|
|
"${cfg}" "${ext}" "${template}"
|
|
|
|
done
|
|
|
|
fi
|
2020-09-06 22:12:22 +02:00
|
|
|
}
|
|
|
|
|
2020-09-06 22:12:24 +02:00
|
|
|
gen_tests() {
|
|
|
|
local -a tests
|
|
|
|
local run_tests tst
|
|
|
|
|
|
|
|
tests=( $(./support/testing/run-tests -l 2>&1 \
|
|
|
|
| sed -r -e '/^test_run \((.*)\).*/!d; s//\1/'\
|
|
|
|
| LC_ALL=C sort)
|
|
|
|
)
|
|
|
|
|
|
|
|
run_tests=false
|
|
|
|
if [ -n "${CI_COMMIT_TAG}" ]; then
|
|
|
|
# For tags, create a pipeline.
|
|
|
|
run_tests=true
|
|
|
|
fi
|
|
|
|
if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
|
|
|
|
# For pipeline created by using a trigger token.
|
|
|
|
run_tests=true
|
|
|
|
fi
|
|
|
|
case "${CI_COMMIT_REF_NAME}" in
|
|
|
|
# For the branch or tag name named *-runtime-tests, create a pipeline.
|
2020-09-06 22:12:26 +02:00
|
|
|
(*-runtime-tests)
|
|
|
|
run_tests=true
|
|
|
|
;;
|
|
|
|
(*-tests.*)
|
|
|
|
tests=( "${CI_COMMIT_REF_NAME##*-}" )
|
|
|
|
run_tests=true
|
|
|
|
;;
|
2020-09-06 22:12:24 +02:00
|
|
|
esac
|
|
|
|
|
|
|
|
if ${run_tests}; then
|
2020-09-06 22:12:27 +02:00
|
|
|
printf '%s: { extends: .runtime_test_base }\n' "${tests[@]}"
|
2020-09-06 22:12:24 +02:00
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
2020-09-06 22:12:20 +02:00
|
|
|
main "${@}"
|