2020-12-19 16:35:20 +01:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
|
|
|
|
set -e
|
|
|
|
|
|
|
|
. "${0%/*}/helpers"
|
|
|
|
|
|
|
|
while getopts "n:o:" OPT; do
|
|
|
|
case "${OPT}" in
|
|
|
|
o) output="${OPTARG}";;
|
|
|
|
n) base_name="${OPTARG}";;
|
|
|
|
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
|
|
|
|
\?) error "unknown option '%s'\n" "${OPTARG}";;
|
|
|
|
esac
|
|
|
|
done
|
|
|
|
|
|
|
|
# Already vendored tarball, nothing to do
|
|
|
|
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
|
|
|
|
exit 0
|
|
|
|
fi
|
|
|
|
|
|
|
|
post_process_unpack "${base_name}" "${output}"
|
|
|
|
|
|
|
|
# Do the Cargo vendoring
|
|
|
|
pushd "${base_name}" > /dev/null
|
|
|
|
|
|
|
|
# Create the local .cargo/config with vendor info
|
|
|
|
mkdir -p .cargo/
|
support/download: fix concurrent cargo vendor
Commit 8450b7691870 (package/pkg-cargo: move CARGO_HOME into DL_DIR)
allowed for a shared cargo cache of crates. Internally, cargo is
supposed to lock themselves when accessing that cache, and that commit
even had some research in that area, pointing at [0] for complaints
about too-coarse the lock, so it was deemed safe to have a shared cargo
home.
However, in practice, the locking as implemented by cargo, fails to
properly protect the concurrent accesses to the crates cache, with random
failures that manifest themselves like so:
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Downloading crates ...
error: failed to sync
Caused by:
failed to download packages
Caused by:
failed to download `autocfg v1.1.0`
Caused by:
unable to get packages from source
Caused by:
failed to unpack package `autocfg v1.1.0`
Caused by:
failed to unpack entry at `autocfg-1.1.0/src/tests.rs`
Caused by:
No such file or directory (os error 2) while canonicalizing [...]
with the last few errors sometime being:
Caused by:
failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml`
Caused by:
can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path
So, as we do not systematically use our own cargo build (we can use a
pre-built one with host-rust-bin), we can't patch cargo (even if we knew
what to do!).
Instead, we implement a lock ourselves, by wrapping the call to "cargo
vendor" with a flock(1) on cargo home.
Note: the download wrapper is already flock-ed, but it is a per-package
lock, so it does not prevent different packages from being downloaded in
parallel; if those packages need cargo vendoring, that will not be
protected by the flock on the dl wrapper. So we really do need a flock
on cargo home.
[0] https://github.com/rust-lang/cargo/issues/6930
Fixes: 8450b769187087751f83cbefcf0a88f70d9da670
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Moritz Bitsch <moritz@h6t.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-01-14 15:48:53 +01:00
|
|
|
mkdir -p "${CARGO_HOME}"
|
|
|
|
flock "${CARGO_HOME}"/.br-lock \
|
2022-10-25 12:02:00 +02:00
|
|
|
cargo vendor \
|
|
|
|
--manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
|
|
|
|
--locked VENDOR \
|
support/download: fix the cargo post-process in face of failed vendoring
In commit 04154a651729 (support/download/cargo-post-process: cargo
output for vendor config), we switched away from our hand-crafted
cargo.toml mangling, to use cargo itself to update that file.
In doing so, we enabled the shell pipefail option, so that we could
catch cargo failures, while redirecting its output through tee to the
cargo.toml.
However, pipefail is overzealous, and will hit us even for pipes we do
not want to globally fail, like the one that actually checks whether an
archive is already vendored or not:
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
...
with pipefail, the above may always fail:
- if the tarball is already vendored, grep will exit on the first
match because of -q (it only needs a single match to decide that its
return code will be zero), so the | will get closed, and tar may
get -EPIPE before it had a chance to finish listing the archive, and
thus would terminate in error;
- if the tarball is not vendored, grep will exit in error.
It turns out that the tee was only added so that we could see the
messages emitted by cargo, and still fill the cargo.tom with the output
of cargo.
But that's a bit overkill: the cargo messages are going to stderr, and
the blurb to add to cargo.toml to stdout, so we just need to redirect
stdout.
Yes, we do not see what cargo added to cargo.toml, but that is not so
interesting.
Still, cargo ends its messages with a suggestion for the user to modify
cargo.toml, with:
To use vendored sources, add this to your .cargo/config.toml for this project:
But since we've already redirected that to cargo.toml, there is nothing
for the user to edit, so the above can get confusing. Emit a little
blurb that states that everything is under control.
And then we can drop pipefail.
Note: the go-post-process initially had pipefail too, but it was dropped
in bfd1a31d0e59 (support/download/go-post-process: drop -o pipefail) as
it was causing spurious breakage when extracting the archive before
vendoring, so it is only reasonable that we also remove it from the
cargo-post-process.
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-10 23:31:43 +01:00
|
|
|
> .cargo/config
|
|
|
|
|
|
|
|
# "cargo vendor' outputs on stderr a message directing to add some data
|
|
|
|
# to the project's .cargo/config.toml, data that it outputs on stdout.
|
|
|
|
# Since we redirect stdout to .cargo/config.toml, the message on stderr
|
|
|
|
# gets confusing, so instruct the user that it's been handled.
|
|
|
|
printf '(note: .cargo/config.toml automatically updated by Buildroot)\n\n'
|
2020-12-19 16:35:20 +01:00
|
|
|
|
|
|
|
popd > /dev/null
|
|
|
|
|
|
|
|
post_process_repack "$(pwd)" "${base_name}" "${output}"
|