kumquat-buildroot/support/scripts/generate-gitlab-ci-yml

#!/usr/bin/env bash
set -e
set -o pipefail

input="${1}"

cat "${input}"

(
    cd configs
    LC_ALL=C ls -1 *_defconfig
) \
    | sed -r -e 's/^(.+)$/\1: { extends: .defconfig }\n\1_check: { extends: .defconfig_check }/'

./support/testing/run-tests -l 2>&1 \
    | sed -r -e '/^test_run \((.*)\).*/!d; s//\1: { extends: .runtime_test }/' \
    | LC_ALL=C sort
Makefile: offload .gitlab-ci.yml generation GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: \| only: \| - /-$CI_JOB_NAME$/ \| - /-\$CI_JOB_NAME$/ \| - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: \| only: \| refs: \| - branches \| - tags \| variables: \| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: \|.runtime_test: &runtime_test \| only: \| - tags \|tests.package.test_python_txaio.TestPythonPy2Txaio: \| <<: *runtime_test \| only: \| - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> 2018-10-29 00:58:38 +01:00			`#!/usr/bin/env bash`
			`set -e`
			`set -o pipefail`

			`input="${1}"`

			`cat "${input}"`

			`(`
			`cd configs`
			`LC_ALL=C ls -1 *_defconfig`
			`) \`
gitlab-ci: check all defconfigs on every push The defconfig check has been introduced by the previous patch before the building each defconfig but those builds are done every week or more. Checking if a defconfig is valid can be done on every push in the repository since it take few seconds. This would allow to detect as soon as possible a problem in a defconfig and eventually avoid breaking the build while build testing all defconfig. Introduce a new job template ".defconfig_check" in gitlab-ci.yml.in and modify the generate-gitlab-ci-yml to create a job for each defconfig to run the test. Although, we could have used only one job to do all tests, using one job per defconfig allow to identify easily in gitlab which defconfig is falling. Tested: https://gitlab.com/kubu93/buildroot/pipelines/138331069 https://gitlab.com/kubu93/buildroot/pipelines/171223758 Signed-off-by: Romain Naour <romain.naour@gmail.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> 2020-07-27 17:51:27 +02:00			`\| sed -r -e 's/^(.+)$/\1: { extends: .defconfig }\n\1_check: { extends: .defconfig_check }/'`
Makefile: offload .gitlab-ci.yml generation GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: \| only: \| - /-$CI_JOB_NAME$/ \| - /-\$CI_JOB_NAME$/ \| - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: \| only: \| refs: \| - branches \| - tags \| variables: \| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: \|.runtime_test: &runtime_test \| only: \| - tags \|tests.package.test_python_txaio.TestPythonPy2Txaio: \| <<: *runtime_test \| only: \| - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> 2018-10-29 00:58:38 +01:00
			`./support/testing/run-tests -l 2>&1 \`
.gitlab-ci.yml: use "extends" keyword Replace all YAML anchors with the new "extends" keyword because it is more readable and more flexible (it works across configuration files combined with the new "include" keyword). Readability is more meaningful in .gitlab-ci.yml.in. In the part of .gitlab-ci.yml that is auto-generated by 'make .gitlab-ci.yml' keep the keyword in the same line of the job name. So instead of this: zynqmp_zcu106_defconfig: extends: .defconfig tests.boot.test_atf.TestATFAllwinner: extends: .runtime_test Use this: zynqmp_zcu106_defconfig: { extends: .defconfig } tests.boot.test_atf.TestATFAllwinner: { extends: .runtime_test } Do this to to keep .gitlab-ci.yml easier to be post-processed by a script. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> 2019-01-21 02:11:11 +01:00			`\| sed -r -e '/^test_run \((.)\)./!d; s//\1: { extends: .runtime_test }/' \`
Makefile: offload .gitlab-ci.yml generation GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: \| only: \| - /-$CI_JOB_NAME$/ \| - /-\$CI_JOB_NAME$/ \| - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: \| only: \| refs: \| - branches \| - tags \| variables: \| - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: \|.runtime_test: &runtime_test \| only: \| - tags \|tests.package.test_python_txaio.TestPythonPy2Txaio: \| <<: *runtime_test \| only: \| - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> 2018-10-29 00:58:38 +01:00			`\| LC_ALL=C sort`