kumquat-buildroot/support/download/cargo-post-process

44 lines
1.2 KiB
Plaintext
Raw Normal View History

#!/usr/bin/env bash
set -e
. "${0%/*}/helpers"
while getopts "n:o:" OPT; do
case "${OPT}" in
o) output="${OPTARG}";;
n) base_name="${OPTARG}";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
done
# Already vendored tarball, nothing to do
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
exit 0
fi
post_process_unpack "${base_name}" "${output}"
# Do the Cargo vendoring
pushd "${base_name}" > /dev/null
# Create the local .cargo/config with vendor info
mkdir -p .cargo/
support/download: fix concurrent cargo vendor Commit 8450b7691870 (package/pkg-cargo: move CARGO_HOME into DL_DIR) allowed for a shared cargo cache of crates. Internally, cargo is supposed to lock themselves when accessing that cache, and that commit even had some research in that area, pointing at [0] for complaints about too-coarse the lock, so it was deemed safe to have a shared cargo home. However, in practice, the locking as implemented by cargo, fails to properly protect the concurrent accesses to the crates cache, with random failures that manifest themselves like so: Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Downloading crates ... error: failed to sync Caused by: failed to download packages Caused by: failed to download `autocfg v1.1.0` Caused by: unable to get packages from source Caused by: failed to unpack package `autocfg v1.1.0` Caused by: failed to unpack entry at `autocfg-1.1.0/src/tests.rs` Caused by: No such file or directory (os error 2) while canonicalizing [...] with the last few errors sometime being: Caused by: failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml` Caused by: can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path So, as we do not systematically use our own cargo build (we can use a pre-built one with host-rust-bin), we can't patch cargo (even if we knew what to do!). Instead, we implement a lock ourselves, by wrapping the call to "cargo vendor" with a flock(1) on cargo home. Note: the download wrapper is already flock-ed, but it is a per-package lock, so it does not prevent different packages from being downloaded in parallel; if those packages need cargo vendoring, that will not be protected by the flock on the dl wrapper. So we really do need a flock on cargo home. [0] https://github.com/rust-lang/cargo/issues/6930 Fixes: 8450b769187087751f83cbefcf0a88f70d9da670 Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Moritz Bitsch <moritz@h6t.eu> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-01-14 15:48:53 +01:00
mkdir -p "${CARGO_HOME}"
flock "${CARGO_HOME}"/.br-lock \
cargo vendor \
--manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
--locked VENDOR \
support/download: fix the cargo post-process in face of failed vendoring In commit 04154a651729 (support/download/cargo-post-process: cargo output for vendor config), we switched away from our hand-crafted cargo.toml mangling, to use cargo itself to update that file. In doing so, we enabled the shell pipefail option, so that we could catch cargo failures, while redirecting its output through tee to the cargo.toml. However, pipefail is overzealous, and will hit us even for pipes we do not want to globally fail, like the one that actually checks whether an archive is already vendored or not: if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then ... with pipefail, the above may always fail: - if the tarball is already vendored, grep will exit on the first match because of -q (it only needs a single match to decide that its return code will be zero), so the | will get closed, and tar may get -EPIPE before it had a chance to finish listing the archive, and thus would terminate in error; - if the tarball is not vendored, grep will exit in error. It turns out that the tee was only added so that we could see the messages emitted by cargo, and still fill the cargo.tom with the output of cargo. But that's a bit overkill: the cargo messages are going to stderr, and the blurb to add to cargo.toml to stdout, so we just need to redirect stdout. Yes, we do not see what cargo added to cargo.toml, but that is not so interesting. Still, cargo ends its messages with a suggestion for the user to modify cargo.toml, with: To use vendored sources, add this to your .cargo/config.toml for this project: But since we've already redirected that to cargo.toml, there is nothing for the user to edit, so the above can get confusing. Emit a little blurb that states that everything is under control. And then we can drop pipefail. Note: the go-post-process initially had pipefail too, but it was dropped in bfd1a31d0e59 (support/download/go-post-process: drop -o pipefail) as it was causing spurious breakage when extracting the archive before vendoring, so it is only reasonable that we also remove it from the cargo-post-process. Reported-by: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Simon Richter <simon.richter@ptwdosimetry.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-02-10 23:31:43 +01:00
> .cargo/config
# "cargo vendor' outputs on stderr a message directing to add some data
# to the project's .cargo/config.toml, data that it outputs on stdout.
# Since we redirect stdout to .cargo/config.toml, the message on stderr
# gets confusing, so instruct the user that it's been handled.
printf '(note: .cargo/config.toml automatically updated by Buildroot)\n\n'
popd > /dev/null
post_process_repack "$(pwd)" "${base_name}" "${output}"