Switch to using the tarball helper, that can generate reproducible
archives whatever the tar version >= 1.27.
However, those archives are not identical to the previous ones generated
in the (now-broken) gnu format.
To avoid any clashing between old and new archives, and new and old
Buildroot versions, we need to name the new generated archives
differently from the existing ones.
So, we bump the git-specific format-version to -br1.
The %ci date has been supported by git back to 1.6.0, released August
2008); it is not strictly ISO8601, but is still accepted as a PAX date
header. The strict ISO8601 placeholder, %cI, was only introduced with
2.2.0, release in November 2014, so too recent to be widely available.
As the format and the names of the archives changes, we need to update
all the hash files with the new names and hashes.
Of all the bootloaders that have a git download method, vexpress-firmware
is the only one to have a hash. Others have no hash files, or they have
explicitly set BR_NO_CHECK_HASH_FOR.
For the packages, linux-headers is the special snowflake, as the git
download is only for custom git tree, so it is excluded from the hash
verification with BR_NO_CHECK_HASH_FOR.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
---8<------8<------8<------8<---
#!/bin/sh
# Find and download all packages using git as backend.
# Manually fix hashes for affected packages.
# Packages that only have a host variant
HOST_ONLY='imx-mkimage|mxsldr|netsurf-buildsystem|opkg-utils|prelink-cross|qoriq-rcw|vboot-utils'
# Packages that have a non-git main _SOURCE, and/or which
# have BR_NO_CHECK_HASH_FOR for the git _SOURCE
NOT_GIT='aufs|aufs-util|xenomai|linux-headers'
export BR2_DL_DIR=$(pwd)/temp-dl-dir
make defconfig
make $( git grep -l -E 'SITE_METHOD[[:space:]]*:?=[[:space:]]*git\>|_SITE[[:space:]]*:?=[[:space:]]*git:' \
boot/vexpress-firmware/ package/ \
|sed -r -e 's,.*/([^/]+)\.mk,\1,' \
|sed -r -e '/^('"${NOT_GIT}"')$/d;' \
-e 's/^('"${HOST_ONLY}"')/host-\1/;' \
-e 's/$/-legal-info/;'
)
---8<------8<------8<------8<---
Python 2.7 will not be maintained past 2020.
Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.
An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.
In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output
file as binary;
- when subprocess is used to retrieve the text output from commands,
explicitly ask for text output. For this, use 'universal_newlines'
because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git,
explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation
for the Python 2 module, change it to follow the example in the
documentation for the Python 3 module;
- fix the relative import for test_git.py to be Python 3 compliant.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This reverts commit 81771cfcdc.
The download of sha1 of a special ref currently works or not depending
on the git client version in use.
With git version 2.11.0 (present in the docker image) it does not work.
With git version 2.17.1 it works.
For the sake of reproducibility, remove this part of the TestGitRefs
test case until some code gets added to the download infra to handle
sha1 of a special ref for any git client version.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/158295269
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a special ref to the static repo and check on the git refs test case
the download of a git package:
- with the sha1 of a special ref as version;
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: change to use the sha1 of a special ref instead of the name]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a tag to the static repo and check on the git refs test case the
download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Enables the test to use the new non-emulator base class which takes
significantly less test time.
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add two submodules as static repos, add a branch to the main static repo
and check on the git refs test case the download of a git package:
- repo with submodule but without support in the package;
- repo with recursive submodules with support in the package.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: remove handling of inconsistent tarball hashes - that's an
actual bug that should be fixed]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add a branch to the static repo and check on the git refs test case the
download of a git package:
- with a sha1 reachable by a branch name, but not pointed by it, as
version. This is the most common use case for git refs in the tree;
- with a partial sha1 of a commit reachable by a branch as version;
- with a sha1 of the commit head of a branch as version;
- with a partial sha1 of the commit head of a branch as version;
Enforce the download always occurs by removing the BR2_DL_DIR used for
the tarballs generated by the git download infra.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
All upcoming tests for git refs will rely on the return code of make to
determine whether a git ref can be downloaded or not and also to
determine whether the downloaded content is correct (all of this taking
advantage of the check-hash mechanism already in place for git
packages).
So to avoid false results i.e. in the case the check-hash mechanism
become broken in the master branch, add some sanity checks before the
actual test of download git refs.
Add the minimum test case for git refs containing only sanity checks.
Reuse the commit in the static repo.
Add a br2-external with two packages to check that:
- trying to download an invalid sha1 generates an error;
- downloading a valid sha1 that contains unexpected content generates
an error.
In order to ease the maintenance and review, each upcoming patch adding
checks to this test case will add at same time the commits to the static
repo, the equivalent packages to the br2-external and code to the test
case.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add one test case to ensure the hash is checked for git packages:
- correct hash;
- wrong hash;
- no hash file.
Add required infra:
- a GitRemote class, that can start a git server in the host machine to
emulate a remote git server under the control of the test;
- a new base class, called GitTestBase, that inherits from BRTest and
must be subclassed by all git test cases.
Its setUp() method takes care of configuring the build with a
br2-external, avoiding to hit http://sources.buildroot.net by using
an empty BR2_BACKUP_SITE. It also avoids downloading not
pre-installed dependencies (i.e. lzip) every time by calling 'make
dependencies' using the common dl directory, and it instantiates the
GitRemote object.
Besides the Python scripts, add some fixtures used during the tests:
- a br2-external (git-hash) with one package for each part of the test
case;
- a static git bare repo (repo.git) to be served using GitRemote class.
Neither the br2-external nor the check hash functionalities are the
subject of these tests per se, so for simplicity limit the check to the
error codes and don't look for the messages in the log.
Thanks to Arnout for the hint about how to add a bare repo to test.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Arnout: split long line; reorder imports to satisfy flake8]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
