Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.
This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.
Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash
file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the
legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to
re-order the arguments passed to support/download/check-hash, which also
impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infra
That in turn also requires that the legal-license-file macro args get
re-ordered to have the hash file last; we take the opportunity to also
move the HOST/TARGET arg to be first, like in the other legal-info
macros.
Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The quoting around the expansion of ${relative_dir} was indeed incorrect
since it was introduced back in 8fe9894f65 (suport/download: fix git
wrapper with submodules on older git versions): it is in fact already
quoted as part of the whole sed expression.
${GIT} can contain more than one item, but we don't care about splitting
on spaces when we just print it for debug, so we can just quote it
rather than add an exception.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since commit b7efb43e86 (download/git: try to recover from
utterly-broken repositories), we catch errors through an ERR
trap, so we can try and recover from a broken repository. In
that commit, we switched from using "set -e" to "set -E", so
that trap is inherited in functions, command substitutions,
and subshells.
However, the trap is not defined until we have parsed the
options, created the cache directory, and eventually chdir()ed
into it. Athough improbable, it is possible for the git helper
to fail in any of those steps, and that would not get caught.
Fix that
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we generate the taballs off a local working copy of a VCS tree,
the umask is the one that we enforce in out top-level Makefile.
However, it is possible that a user manually tinkers in said working
copy (e.g. to check an upstream bug fix, or regression). If the user
umask is different from the one Buildroot enfirces, such tinkering
can impact the mode bits of the files, even if their content is not
modified.
When we eventually need to create a tarball from said working copy,
the VCS (e.g. git) will only be interested in checking whether the
content of the files have changed before chcking them out, and will
not look at, and restore/fix the mode bits.
As a consequence, we may create non-reproducible archives.
We fix that by enforcing the mode bits on the files before we create
the tarball: we disable the write and execute bits, and only set the
execute bit if the user execute bit is set.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Like git which can have submodules, subversion can have externals. The
default behaviour for subversion is to retrieve all the externals,
unless told otherwise.
For some repositories, the externals may be huge (e.g. a dataset or some
assets) and may not be required for building the package. In such a
case, retrieving the externals is both a waste of network bandwitdh and
time, and a waste of disk storage.
Like for git submodules and git lfs, add an option that packages can set
to specify whether they want externals or not.
Since we've so far been retrieving externals, we keep that the default,
and packages can opt-out (rather than the opt-in for git submodules or
git lfs).
We must only set it when the package is actually hosted on svn, to avoid
passing -r when the package is not hosted by svn; otherwise, -r would
also be passed e.g. to a git-hosted package, triggering the download of
git submodules even when they are not requested. We need to do so,
because we have a default value, which we usually do not have in other
download options.
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When an svn repository requires credentials, and they are passed
in _DL_OPTS, they must be used also to retrieve the revision date.
One could argue that credentials should not be handled in _DL_OPTS, but
rather that they be fed through other means (e.g. by pre-authenticating
manually once in an interactive session, or by filling them in the usual
~/svn/auth/* mechanisms for a CI).
However, some public facing repositories are using authentication, even
though the credentials are public. This is the case for example for:
http://software.rtcm-ntrip.org/
In such a case, it does make sense to pass credentials via _DL_OPTS,
because they are not really, even really not, secret.
Another use-case (e.g. for a CI) is to pass the credentials as
environment variables, with _DL_OPTS not hard-coded in the .mk file.
However, _DL_OPTS may contain options that are not valid for 'svn info',
as they are meant to be passed to 'svn export' in the first place. Since
the only options common to 'svn info' and 'svn export' are the
credentials, we just extract those and pass them to 'svn info'.
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Bizarrely enough, the unquoted expansion of ${quiet} does not trigger
any warning from shellcheck, so we do not add any exception for it.
${SVN} can contain more than one item, but we don't care about splitting
on spaces when we just print it for debug, so we can just quote it
rather than add an exception.
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In commit 04154a6517 (support/download/cargo-post-process: cargo
output for vendor config), we switched away from our hand-crafted
cargo.toml mangling, to use cargo itself to update that file.
In doing so, we enabled the shell pipefail option, so that we could
catch cargo failures, while redirecting its output through tee to the
cargo.toml.
However, pipefail is overzealous, and will hit us even for pipes we do
not want to globally fail, like the one that actually checks whether an
archive is already vendored or not:
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
...
with pipefail, the above may always fail:
- if the tarball is already vendored, grep will exit on the first
match because of -q (it only needs a single match to decide that its
return code will be zero), so the | will get closed, and tar may
get -EPIPE before it had a chance to finish listing the archive, and
thus would terminate in error;
- if the tarball is not vendored, grep will exit in error.
It turns out that the tee was only added so that we could see the
messages emitted by cargo, and still fill the cargo.tom with the output
of cargo.
But that's a bit overkill: the cargo messages are going to stderr, and
the blurb to add to cargo.toml to stdout, so we just need to redirect
stdout.
Yes, we do not see what cargo added to cargo.toml, but that is not so
interesting.
Still, cargo ends its messages with a suggestion for the user to modify
cargo.toml, with:
To use vendored sources, add this to your .cargo/config.toml for this project:
But since we've already redirected that to cargo.toml, there is nothing
for the user to edit, so the above can get confusing. Emit a little
blurb that states that everything is under control.
And then we can drop pipefail.
Note: the go-post-process initially had pipefail too, but it was dropped
in bfd1a31d0e (support/download/go-post-process: drop -o pipefail) as
it was causing spurious breakage when extracting the archive before
vendoring, so it is only reasonable that we also remove it from the
cargo-post-process.
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/12a/12a63ae177fe3ed0c9a1ef2fa01870f334f36b0f/
Currently, when the post-process helper fails while downloading from
upstream, there is no fallback to the backup mirror.
In case the post-process helper fails, we must consider that to be a
download failure, so we must bail out as if the download backend itself
did fail, but we fail to do so.
Duplicate the logic we have for the download helper: if the post-process
helper fails, remove the downloaded stuff, and continue on to the next
URI, which will ultimately hit the backup mirror (if one has been
configured).
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 8450b76918 (package/pkg-cargo: move CARGO_HOME into DL_DIR)
allowed for a shared cargo cache of crates. Internally, cargo is
supposed to lock themselves when accessing that cache, and that commit
even had some research in that area, pointing at [0] for complaints
about too-coarse the lock, so it was deemed safe to have a shared cargo
home.
However, in practice, the locking as implemented by cargo, fails to
properly protect the concurrent accesses to the crates cache, with random
failures that manifest themselves like so:
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Downloading crates ...
error: failed to sync
Caused by:
failed to download packages
Caused by:
failed to download `autocfg v1.1.0`
Caused by:
unable to get packages from source
Caused by:
failed to unpack package `autocfg v1.1.0`
Caused by:
failed to unpack entry at `autocfg-1.1.0/src/tests.rs`
Caused by:
No such file or directory (os error 2) while canonicalizing [...]
with the last few errors sometime being:
Caused by:
failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml`
Caused by:
can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path
So, as we do not systematically use our own cargo build (we can use a
pre-built one with host-rust-bin), we can't patch cargo (even if we knew
what to do!).
Instead, we implement a lock ourselves, by wrapping the call to "cargo
vendor" with a flock(1) on cargo home.
Note: the download wrapper is already flock-ed, but it is a per-package
lock, so it does not prevent different packages from being downloaded in
parallel; if those packages need cargo vendoring, that will not be
protected by the flock on the dl wrapper. So we really do need a flock
on cargo home.
[0] https://github.com/rust-lang/cargo/issues/6930
Fixes: 8450b76918
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Moritz Bitsch <moritz@h6t.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit de4cf25375 (package/{rust, rust-bin}: bump to version 1.66.0)
forgot, despite the big comment above the version strings, to confirm
that the vendoring was still working.
Previously, we were adding the vendoring equivalence manually, but in
commit 04154a6517 (support/download/cargo-post-process: cargo output
for vendor config), we switched to using the output of "cargo vendor"
(on stdout) to support cases were the vendoring equivalence would be
more complex (e.g. when using crates not hosted on crates.io).
With rust until and including 1.65.0, "cargo vendor" would output (for
crates.io crates) the same output as our manual fixups, except it was
preceded by an empty line. So, to avoid recompting all our hashes, we
added a tweak to strip away the leading empty line in 04154a6517.
But rust 1.66.0 includes [0] which changes the output (on stdout) of
"cargo vendor", where the first empty line is no longer emitted.
This means that our tweak for rust 1.65.0 now strips out an important
part of the cargo vendor output, which renders the archives invalid, and
thus generates different archives, which fail to validate against our
hashes.
Fix this by doing what the comment in the post-process helper states,
and just keep the whole output of "cargo vendor", by just removing the
"tail --lines=+2". Since that comment is no longer meaningful, we drop
it too.
Now, all our 6 cargo-based packages, as well as our 5 python packages
that have rust code, can be vendored again, without changing our hashes,
but most importantly, with valid archives.
Still, we keep the comment above the versions strings, in the hope that
a future bumper will notice and be more careful at validating the
vendoring.
[0] https://github.com/rust-lang/cargo/pull/11273
Fixes:
http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
... and so many others...
Reported-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: James Hilliard <james.hilliard1@gmail.com>
Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Reviewed-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Use the output of `cargo vendor` to generate the vendor configuration.
Fixes the need to patch the generated configuration if there are
non-crates.io dependencies.
Note:
`cargo vendor` currently prints a newline before it prints the
needed configuration.
This is fixed in +nightly, will end up in +stable soon and must
be considered when updating cargo.
See: https://github.com/rust-lang/cargo/pull/11273
Until then it is needed to remove this first line to make sure
that the contents of .cargo/config will be the same as they were
generated with the earlier version of the script. Thus, the
hashes of the packages that use this script remain the same.
Signed-off-by: Simon Richter <simon.richter@ptwdosimetry.com>
[yann.morin.1998@free.fr: add comment in rust-bin.mk and rust.mk]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The -z option for head was only added in coreutils 8.25, but some older
enterprise-grade distributions (e.g. the oldest still maintained RHEL 7)
only have nothing more recent than coreutils 8.22.
We fix that by using sed to remove everything that starts with the first
NULL byte, \x00.
Signed-off-by: Clayton Shotwell <clayton.shotwell@collins.com>
[yann.morin.1998@free.fr: hex is \xHH, not \xH, reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 1ba85b7f87 (support/download: add explicit no-hash support)
introduced the 'none' hash type, in an attempt to make hash files
mandatory, but not failing on archives localy generated, like those
for git or svn repositories, especially for those packages where a
version choice was present, which would allow for either remote
archives for which we'd have a hash or VCS trees for which we could
not have a hash for the localy generated archive.
Indeed, back in the time, we did not have a mean to generate
reproducible archives, so having a hash file without a hash for
thosel ocally generated archives would trigger an error in the
hash-checking machinery.
But now, low-and-behold, we do know how to generate those archives,
and we have a mechanism to explicitly exclude some archives from being
hash-checked (e.g. when the version string itself can be user-provided).
As such, the 'none' hash type no longer has any raison d'être, we do not
use it in-tree, and its use in a br2-external tree is most probably
inexistent (as is the use of hash files alotgether most probably).
So we simply drop the support for that.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Thomas: drop support in checkpackagelib, as reported by Ricardo.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In most pure Rust packages, the Cargo.toml manifest is at the root
directory, which is why we could call "cargo vendor" without
specifying the path of the manifest.
However, other packages, such as python-cryptography, which have parts
implemented in Rust, have their Cargo.toml located in a specific
subdirectory.
This commit extends the cargo-post-process download script to
understand a BR_CARGO_MANIFEST_PATH environment variable, which allows
a package to pass the location of the Cargo.toml file. If not passed,
"Cargo.toml" is used, preserving the existing behavior for other
packages.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This breaks the post_process_unpack() function in
support/download/helpers, which had a sequence of pipe, with "head"
that can abort early and cause the pipe to fail.
Fixes intermitent:
make[1]: *** [package/pkg-generic.mk:190: /builds/tpetazzoni/buildroot/test-output/TestDockerCompose/build/containerd-1.5.8/.stamp_downloaded] Error 141
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In order to be package agnostic, the install phase is now using cargo
instead of install. TARGET_CONFIGURE_OPTS is now also set when running
cargo in order to support cross compiling C code within cargo.
This commit also adds support/download/cargo-post-process to perform
the vendoring on Cargo packages.
The <pkg>_LICENSE variable of cargo packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.
Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: add support for host-cargo-package and vendoring]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.
The -modcacherw flag is added to ensure that the Go cache is
read/write, and can be deleted properly upon "make clean".
The <pkg>_LICENSE variable of golang packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For now, the download post-process logic uses mk_tar_gz, which repacks
a tarball compressed with gzip. So we can only accept as input a
tarball also compressed with gzip. To enforce that, this commit
changes post_process_unpack() to use tar xzf. This makes sure that if
a tarball compressed with something else than gzip gets used, it will
bail out and we will notice.
Support for other compression schemes can be added later on.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The original patch for commit cff428fe31 ("download/git: support Git
LFS") included a call to "git lfs install" but this was a problem as it
could modify ~/.gitconfig outside the dl/ tree. When this was
updated it was thought that the modification to gitconfig was
unnecessary because the LFS fetch and checkout steps are performed
manually.
Unfortunately, this is not correct and the LFS checkout fails with:
Cannot checkout LFS objects, Git LFS is not installed.
Add the call to "git lfs install", with the --local option so that only
the repository's .git/config is modified and not the user's global
~/.gitconfig.
This is also required for submodules as the parent repository's config
is not inherited.
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: John Keeping <john@metanate.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.
To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- double-quote variable expansion when calling post-process script
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Git Large File Storage replaces large files with text pointers in the
Git repository while storing the contents on a remote server. If a
repository is using this extension, then git-lfs must be used to
checkout the large files before the source archive is generated.
Signed-off-by: John Keeping <john@metanate.com>
[vfazio:
- add git-lfs to DL_TOOLS_DEPENDENCIES
- fixup for 5a0d681394
("infra/pkg-download: make the DOWNLOAD macro fully parameterised")
]
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[Arnout:
- don't "git lfs install";
- recurse into submodules.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add Secure File Transfer Program (SFTP) support using a simple wrapper.
SFTP is a common protocol used to transfer files securely between
enterprises, but it is not currently supported in Buildroot because all
of the packages are usually available via HTTP, git or some other
download method.
SFTP is similar to FTP but it preforms all operations over an encrypted
SSH transport using a specific protocol. This is unlike ftps, which is
traditional FTP over an SSL/TLS connection.
Signed-off-by: Thomas Preston <thomas.preston@codethink.co.uk>
Signed-off-by: Michael Drake <michael.drake@codethink.co.uk>
[Arnout:
- update documentation with sftp everywhere scp is mentioned;
- rename "verbose" variable to "quiet";
- print the sftp command, similar to wget and scp helpers.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Commit 54d3d94b6e broke the 'hg' download
method, in a similar way as it broke the 'git' download method (later fixed
with commit b70ce56651), by introducing extra
output on stdout in a case where the output is redirected.
In the case of 'hg', the 'hg archive' step uses shell redirection rather
than directly letting hg write the output file, since commit
76b51f90c0.
As a result, the extra print added by the _hg function is prepended to the
actual archive, causing an invalid archive.
Fix by using the _plain_hg function instead. The disadvantage is that the
command for 'hg archive' is no longer printed.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When a --transform expression is provided, it is by default also applied
to the target of a symlink.
When we create tarballs (from git or svn checkouts), we use a --transform
expression to replace the leading ./ with the package name and version.
This causes issues when a package contains symlinks that points to
./something, as the leading './' is also replaced.
Fix that by using the 'S' transformation scope flag, as described in the
tar manual:
https://www.gnu.org/software/tar/manual/html_node/transform.html#transform
In addition, several transformation scope flags are supported, that
control to what files transformations apply. These are:
‘r’ Apply transformation to regular archive members.
‘R’ Do not apply transformation to regular archive members.
‘s’ Apply transformation to symbolic link targets.
‘S’ Do not apply transformation to symbolic link targets.
‘h’ Apply transformation to hard link targets.
‘H’ Do not apply transformation to hard link targets.
Default is ‘rsh’ [...].
Fixes: #13616
This has been checked to not change any of the existing hash for any of
our git-downloaded package (some are host-only, hence the few fixups):
---8<---
$ m="$( git grep -l -E -- -br[[:digit:]]+.tar.gz boot package/ \
|awk -F/ '{print $(NF-1)}' \
|sed -r -e 's/(imx-mkimage|netsurf-buildsystem|prelink-cross|qoriq-rcw|vboot-utils)/host-\1/g' \
-e 's/$/-source/'
)"
$ make defconfig; make clean; BR2_DL_DIR=$(pwd)/trash-me make ${m}
---8<---
Note: it is unclear what the 'H' flag does nor how it works, because the
concept of "target of a hardlink" is not obvious; probably it has to do
with how tar internally detects and stores hardlinks. Since we do not
yet have any issue with hardlinks, just ignore the problem for now, and
postpone until we have an actual issue with a real test-case.
Signed-off-by: Jean-pierre Cartal <jpcartal@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
- re-indent commit log
- add scriptlet to test existing hashes
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 54d3d94b6e ("support/download: print
command used for download") broke the git and svn download helpers, because
these helpers have invocations of the _git/_svn commands where the exact
output matters.
For example for git, this would result in:
date: invalid date ‘GIT_DIR=.../dl/libyuv/git/.git git log -1 --pretty=format:%ci \n2019-04-12 17:48:45 +0000’
Detected a corrupted git cache.
Removing it and starting afresh.
Fix by splitting the _git function in two: _git and _plain_git.
The former echoes the command, and then calls the latter.
Most invocations use _git as before, but those cases where the output should
not be disturbed, directly call _plain_git.
For symmetry, all download helpers are aligned, even though only the git and
svn helpers were broken.
Fixes: #13631
Fixes:
http://autobuild.buildroot.org/results/c2f/c2fcd4aa6660e3c2f9c6f85646ca7dfe0db56040/
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: add bug report and autobuild failure]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Even though that most download commands actually print some output, like
progress indication or other messages, the actual command used is not. This
makes it hard to analyze a build log when you are not fully familiar with
the typical output of said log.
Update the download helpers to do just that, respecting any quiet/verbose
flag so that a silent make (make -s) does not get more verbose.
Note: getting rid of the duplication of the command in the script is not
straightforward without breaking support for arguments with spaces.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[yann.morin.1998@free.fr: use printf, not echo]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Most 'verbose' variable inside the download helpers actually mean 'quiet'.
I.e. they are assigned in case quiet operation is requested, and empty in
case of non-quiet operation. Using the name 'verbose' for such a variable is
confusing, especially when you want to test the variable on emptiness or
non-emptiness (in a subsequent commit).
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Since c043ecb20c (support/download: change format of archives
generated from svn), the svn backend uses the generic helper to
create reproducible archives.
That helper really does its job as expected, but the svn backend
is flawed in two ways:
- the first, most obvious breakage happens with versions older
than 1.9, as they do not support the '--show-item' option
for the 'info' action;
- the second is more involved, in that svn will by default
expand the old, legacy, deprecated, cumbersome CVS-style
keywords, in the form of revision marks like '$Date$' in a
C-style comment in a source file. These replacements are
done on checkout as well as on export, and they use local
settings, like the local locale and timezone.
This means that two people with different settings, will get
different sources when the svn-checkout or svn-export the same
revision from the same tree...
Needless to say that this is not very reproducible...
While the first is easily solved, the second is more involved.
We need to ensure that what source is used initially to compute
the hash, will also be the source that are used to check the hash.
There are basically two solutions:
1. we ensure the same environment, by forcing the timezone and
the locale to arbitrary values
2. we disable keyword expansion
For the first solution, this still leaves the possibility that we
miss some environment settings that have an impact on the keyword
expansion. It would mean that Yann's settings be used, as he did
introduce the hash for the only svn-downloaded package we have,
avrdude, settings which are:
TZ=Europe/Paris
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="fr_FR.utf8"
LC_NUMERIC="fr_FR.utf8"
The second option means that the generated archives change. That
means we'd have to bump the archive version for svn downloads, and
that we update the hashes for all the svn-downloaded packages.
We chose to go with the second option, because this is what really
makes more sense, rather than hard-coding arbitrary values in the
environment. And we also have only one svn-downloaded package,
avrdude.
And thus, we're reaching the trigger for this change: avrdude is
impacted by the CVS-keyword expansion issue:
https://svn.savannah.gnu.org/viewvc/avrdude/trunk/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js?revision=1396&view=markup
which would give two different files when checked out on different
machines:
diff -durN foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js
--- foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200
+++ bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200
@@ -1,6 +1,6 @@
/**
* @preserve jquery.layout 1.3.0 - Release Candidate 30.51
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
* $Rev: 303005 $
*
* Copyright (c) 2012
@@ -4718,7 +4718,7 @@
/**
* jquery.layout.state 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2010
* Kevin Dalman (http://allpro.net)
@@ -5074,7 +5074,7 @@
/**
* jquery.layout.buttons 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2010
* Kevin Dalman (http://allpro.net)
@@ -5356,7 +5356,7 @@
/**
* jquery.layout.browserZoom 1.0
- * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $
+ * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $
*
* Copyright (c) 2012
* Kevin Dalman (http://allpro.net)
So we also update the hash for avrdude.
Fixes:
http://autobuild.buildroot.org/results/e3b/e3b0508047f32008ebfa83c5255ec5994b6af120/ (time issue)
http://autobuild.buildroot.org/results/48e/48e78e84b425e79cdb98c16ab40247a0fa7e9676/ (keyword expansion issue)
Reported-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Alexander Sverdlin <alexander.sverdlin@gmail.com>
Reviewed-by: Alexander Sverdlin <alexander.sverdlin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Some download backends, like svn, will provide timestamps with a
sub-second precision, e.g.
$ svn info --show-item last-changed-date [...]
2021-02-19T20:22:34.889717Z
However, the PAX headers do not accept sub-second precision, leading to
failure to download from subversion:
tar: Time stamp is out of allowed range
tar: Exiting with failure status due to previous errors
make[1]: *** [package/pkg-generic.mk:148: [...]/build/subversion-1886712/.stamp_downloaded] Error 1
Fix that by massaging the timestamp to drop the sub-second part. We
do that in the generic helper, rather than the svn backend, so that
all callers to the generic helper benefit from this, as this is more
an internal details of the tarball limitations, than of the backends
themselves.
Reported-by: Roosen Henri <Henri.Roosen@ginzinger.com>
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
- add Henri as reporter
- move it out of the svn backend, and to the generic helper
- reword the commit log accordingly
- use an explicit time format rather than -Iseconds
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Like we recently did for git, switch the archives generated from
subversion to be reproducible whatever the tar version.
We have no in-tree users of the svn backend which also has hashes,
so no hash to update.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Commit 89f5e9893 (support/download/svn: generate reproducible svn
archives) did what it said, but can be siplified a bit.
Indeed, we are doing an svn export, so we won't have any of the .svn
directories, neither at the root of the extract, nor in any of the
sub-directories.
As such, we do not need to filter them out when we generate the list
of files to include in the archive.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Switch to using the tarball helper, that can generate reproducible
archives whatever the tar version >= 1.27.
However, those archives are not identical to the previous ones generated
in the (now-broken) gnu format.
To avoid any clashing between old and new archives, and new and old
Buildroot versions, we need to name the new generated archives
differently from the existing ones.
So, we bump the git-specific format-version to -br1.
The %ci date has been supported by git back to 1.6.0, released August
2008); it is not strictly ISO8601, but is still accepted as a PAX date
header. The strict ISO8601 placeholder, %cI, was only introduced with
2.2.0, release in November 2014, so too recent to be widely available.
As the format and the names of the archives changes, we need to update
all the hash files with the new names and hashes.
Of all the bootloaders that have a git download method, vexpress-firmware
is the only one to have a hash. Others have no hash files, or they have
explicitly set BR_NO_CHECK_HASH_FOR.
For the packages, linux-headers is the special snowflake, as the git
download is only for custom git tree, so it is excluded from the hash
verification with BR_NO_CHECK_HASH_FOR.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
---8<------8<------8<------8<---
#!/bin/sh
# Find and download all packages using git as backend.
# Manually fix hashes for affected packages.
# Packages that only have a host variant
HOST_ONLY='imx-mkimage|mxsldr|netsurf-buildsystem|opkg-utils|prelink-cross|qoriq-rcw|vboot-utils'
# Packages that have a non-git main _SOURCE, and/or which
# have BR_NO_CHECK_HASH_FOR for the git _SOURCE
NOT_GIT='aufs|aufs-util|xenomai|linux-headers'
export BR2_DL_DIR=$(pwd)/temp-dl-dir
make defconfig
make $( git grep -l -E 'SITE_METHOD[[:space:]]*:?=[[:space:]]*git\>|_SITE[[:space:]]*:?=[[:space:]]*git:' \
boot/vexpress-firmware/ package/ \
|sed -r -e 's,.*/([^/]+)\.mk,\1,' \
|sed -r -e '/^('"${NOT_GIT}"')$/d;' \
-e 's/^('"${HOST_ONLY}"')/host-\1/;' \
-e 's/$/-legal-info/;'
)
---8<------8<------8<------8<---
We currently need to generate reproducible archives in at least two
locations: the git and svn download backends. We also know of some
future potential use (e.g. the other download backends, like cvs, or
in the upcoming download post-processors for vendoring, like cargo
and go).
However, we are currently limited to a narrow range of tar versions
that we support, to create reproducible archives, because the gnu
format we use has changed with tar 1.30.
As a consequence, and as time advances, more and more distros are,
or will eventually start, shipping with tar 1.30 or later, and thus
we need to always build our on host-tar.
Now, thanks to some grunt work by Vincent, we have a set of options
that we can pass tar, to generate reproducible archives back from
tar-1.27 and up through tar-1.32, the latest released version.
However, those options are non-trivial, so we do not want to have
to repeat those (and maintain them) in multiple locations.
Introduce a helper that can generate a reproducible archive from
an input directory.
The --pax-option, to set specific PAX headers, does not accept
RFC2822 timestamps which value are too away from some fixed point
(set atcompile-time?):
tar: Time stamp is out of allowed range
However, the same timestamps passed as strict compliant ISO 8601 are
accepted, so that's what we expect as a date format.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
---8<------8<------8<------8<---
# Here is a Makefile used to test all the versions of tar, with
# different output formats and different sets of options:
# Versions prior to 1.27 do not build on recent machines, because
# 'gets()' got removed (rightfully so), so don't count them as
# candidates.
VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32
DATE = Thu 21 May 2020 06:44:11 PM CEST
TARS = \
$(patsubst %,test_gnu_%.tar,$(VERSIONS)) \
$(patsubst %,test_posix_%.tar,$(VERSIONS)) \
$(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS))
all: $(TARS)
sha1sum $(^)
.INTERMEDIATE: test_%.tar
test_gnu_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=gnu \
-T list \
>$(@)
test_posix_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=posix \
-T list \
>$(@)
test_posix_paxoption_%.tar: tar.% list
./$(<) cf - -C test \
--transform="s#^\./#test-version/#" \
--numeric-owner --owner=0 --group=0 \
--mtime="$(DATE)" \
--format=posix \
--pax-option='delete=atime,delete=ctime,delete=mtime' \
--pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \
-T list \
>$(@)
list: .FORCE
list: test
(cd test && find . -not -type d ) |LC_ALL=C sort >$(@)
LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng
test: .FORCE
test:
rm -rf test
mkdir -p test/bar
echo foo >test/Foo
echo bar >test/bar/Bar
ln -s bar/Bar test/buz
echo long >test/Very-$(LONG)-filename
ln test/Very-$(LONG)-filename \
test/short
.PRECIOUS: tar.%
tar.%: tar-%
cd $(<) && ./configure
$(MAKE) -C $(<)
install -m 0755 $(<)/src/tar $(@)
.PRECIOUS: tar-%
tar-%: tar-%.tar.gz
tar xzf $(<)
.PRECIOUS: tar-%.tar.gz
tar-%.tar.gz:
wget "https://ftp.gnu.org/gnu/tar/$(@)"
.FORCE:
clean:
rm -rf tar-* tar.* test_* test list
---8<------8<------8<------8<---
Older versions of git store the absolute path of the submodules'
repository as stored in the super-project, e.g.:
$ cat some-submodule/.git
gitdir: /path/to/super-project/.git/modules/some-submodule
Obviously, this is not very reproducible.
More recent versions of git, however, store relative paths, which
de-facto makes it reproducible.
Fix older versions by replacing the absolute paths with relative ones.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
To generate a reproducible archive from a svn repository mainly the same
aproach is done like for the archives from a git repository.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[yann.morin.1998@free.fr: get the date of the revision]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This reverts commit 6f35d96756.
Repeat after me: on the master branch you will not work. On the master
branch you will not work.
This definitely shouldn't have been pushed. Sorry about that.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Running "git fetch origin ${cset}:${cset}" to create a local ref
${cset} from the remote ref ${cset} causes Git to issue a warning like
the below, when the version is a full commit hash:
===
warning: refname '49eb4ecb1ef9879ebc6789a1bdb536ab2b1d9871' is ambiguous.
Git normally never creates a ref that ends with 40 hex characters
because it will be ignored when you just specify 40-hex. These refs
may be created by mistake. For example,
git switch -c $br $(git rev-parse ...)
where "$br" is somehow empty and a 40-hex ref is created. Please
examine these refs and maybe delete them. Turn this message off by
running "git config advice.objectNameWarning false"
===
This warning is very confusing for users, and is caused by the fact
that Git doesn't like our local ref name to look like a commit hash.
So, this commit proposes to fix the issue by having the local ref
named buildroot-${cset}, i.e
buildroot-${version-specified-by-the-package}.
The generated tarballs are exactly identical, nothing changes, it is
really just internally the local ref we are using to checkout the
correct version that is different. And it avoids the confusing
warning.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
'.' should be at the end of the sentence, not the beginning of a new
line.
Signed-off-by: John Keeping <john@metanate.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The scp download helper is broken when the server URL starts with 'scp://'.
Such prefix is used in two situations:
1. to let FOO_SITE point to an scp location without explicitly having to set
'FOO_SITE_METHOD = scp'
2. when BR2_PRIMARY_SITE or BR2_BACKUP_SITE points to an scp location. In
this case, there is no equivalent of 'SITE_METHOD'.
Strip out the scheme prefix, similarly to how the 'file' download helper
does it. That helper has the same cases as above.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
scp download is broken, because scp is called without filename argument and
only the server is specified. The call is:
scp <server> <outputfile>
but should be:
scp <server>/<filename> <outputfile>
Instead of assuming '-u' lists a full URL including filename (which it is
not), align with the wget helper where -u is the server URL and -f gives the
filename.
With this commit, an scp download can work if FOO_SITE_METHOD is explicitly
set to 'scp' and the server does not have a scheme prefix 'scp://'.
The next commit will handle the case where a scheme prefix is present.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: s/URL/URI/, as noticed by Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since commit 38de434123 ("download: fix file:// BR2_PRIMARY_SITE
(download cache)"), the urlencode option is no longer passed to the
download backend, because we use ${backend} instead of
${backend_urlencode}.
We must get the urlencode information from backend_urlencode.
Signed-off-by: Damien Thébault <damien.thebault@vitec.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: rework commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This file uses leading spaces, not TABs.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
wget is the only downloader currently usable with BR2_PRIMARY_SITE, and that
doesn't work at all for file:// URLs. The symptoms are these:
support/download/dl-wrapper -c '2.4.47' -d '/PATH/build/sw/source/attr' -D '/PATH/build/sw/source' -f 'attr-2.4.47.src.tar.gz' -H 'package/attr//attr.hash' -n 'attr-2.4.47' -N 'attr' -o '/PATH/build/sw/source/attr/attr-2.4.47.src.tar.gz' -u file\|urlencode+file:///NFS/buildroot_dl_cache/attr -u file\|urlencode+file:///NFS/buildroot_dl_cache -u http+http://download.savannah.gnu.org/releases/attr -u http\|urlencode+http://sources.buildroot.net/attr -u http\|urlencode+http://sources.buildroot.net --
file:///NFS/buildroot_dl_cache/attr/attr-2.4.47.src.tar.gz: Unsupported scheme `file'.
ERROR: attr-2.4.47.src.tar.gz has wrong sha256 hash:
ERROR: expected: 25772f653ac5b2e3ceeb89df50e4688891e21f723c460636548971652af0a859
ERROR: got : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
In the case of custom Linux kernel versions, this is fatal, because there isn't
necessarily a hash file to indicate that wget's empty tarball is wrong.
This seems to have been broken by commit c8ef0c03b0, because:
1. BR2_PRIMARY_SITE always appends "urlencode" (package/pkg-download.mk)
2. Anything with the "|urlencode" suffix in $uri will end up using wget due to
the backend case wildcarding.
3. The wget backend rejects file:/// URLs ("unsupported scheme"), and we end up
with an empty .tar.gz file in the downloads directory.
Fix that by shell-extracting the backend name from the left of "|". I'm not
positive if all URLs will have a "|", so this code only looks for a "|" left of
the "+".
Signed-off-by: Hollis Blanchard <hollis_blanchard@mentor.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The download wrapper is a purely internal helper, and is not supposed to
be callable manually. No need to offer some help.
Besides, the help text was way out-dated.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
'+' is a valid character in a url. The current dl-wrapper gets the
URI scheme by dropping everything after the last '+' character, with
the intension of finding 'git' from e.g. 'git+https://uri'.
If a uri has a '+' anywhere in it, it ends up using too much of the
string as a scheme, and fails to match the handler properly.
An example of where this form of URI is used is when using deploy tokens
in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
where username for deploy token is of the form 'gitlab+deploy-token-<number>'.
Use the %% operator to search backwards until the last '+' character when
dropping the rest of the string as we know that the first '+'
in the string should be the scheme.
Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
