kumquat-buildroot/support/download/cargo-post-process

39 lines
856 B
Plaintext
Raw Normal View History

#!/usr/bin/env bash
set -e
set -o pipefail
. "${0%/*}/helpers"
while getopts "n:o:" OPT; do
case "${OPT}" in
o) output="${OPTARG}";;
n) base_name="${OPTARG}";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
done
# Already vendored tarball, nothing to do
if tar tf "${output}" | grep -q "^[^/]*/VENDOR" ; then
exit 0
fi
post_process_unpack "${base_name}" "${output}"
# Do the Cargo vendoring
pushd "${base_name}" > /dev/null
# Create the local .cargo/config with vendor info
mkdir -p .cargo/
support/download: fix concurrent cargo vendor Commit 8450b7691870 (package/pkg-cargo: move CARGO_HOME into DL_DIR) allowed for a shared cargo cache of crates. Internally, cargo is supposed to lock themselves when accessing that cache, and that commit even had some research in that area, pointing at [0] for complaints about too-coarse the lock, so it was deemed safe to have a shared cargo home. However, in practice, the locking as implemented by cargo, fails to properly protect the concurrent accesses to the crates cache, with random failures that manifest themselves like so: Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Downloading crates ... error: failed to sync Caused by: failed to download packages Caused by: failed to download `autocfg v1.1.0` Caused by: unable to get packages from source Caused by: failed to unpack package `autocfg v1.1.0` Caused by: failed to unpack entry at `autocfg-1.1.0/src/tests.rs` Caused by: No such file or directory (os error 2) while canonicalizing [...] with the last few errors sometime being: Caused by: failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml` Caused by: can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path So, as we do not systematically use our own cargo build (we can use a pre-built one with host-rust-bin), we can't patch cargo (even if we knew what to do!). Instead, we implement a lock ourselves, by wrapping the call to "cargo vendor" with a flock(1) on cargo home. Note: the download wrapper is already flock-ed, but it is a per-package lock, so it does not prevent different packages from being downloaded in parallel; if those packages need cargo vendoring, that will not be protected by the flock on the dl wrapper. So we really do need a flock on cargo home. [0] https://github.com/rust-lang/cargo/issues/6930 Fixes: 8450b769187087751f83cbefcf0a88f70d9da670 Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Moritz Bitsch <moritz@h6t.eu> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-01-14 15:48:53 +01:00
mkdir -p "${CARGO_HOME}"
flock "${CARGO_HOME}"/.br-lock \
cargo vendor \
--manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
--locked VENDOR \
support/download: fix cargo vendoring Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) forgot, despite the big comment above the version strings, to confirm that the vendoring was still working. Previously, we were adding the vendoring equivalence manually, but in commit 04154a651729 (support/download/cargo-post-process: cargo output for vendor config), we switched to using the output of "cargo vendor" (on stdout) to support cases were the vendoring equivalence would be more complex (e.g. when using crates not hosted on crates.io). With rust until and including 1.65.0, "cargo vendor" would output (for crates.io crates) the same output as our manual fixups, except it was preceded by an empty line. So, to avoid recompting all our hashes, we added a tweak to strip away the leading empty line in 04154a651729. But rust 1.66.0 includes [0] which changes the output (on stdout) of "cargo vendor", where the first empty line is no longer emitted. This means that our tweak for rust 1.65.0 now strips out an important part of the cargo vendor output, which renders the archives invalid, and thus generates different archives, which fail to validate against our hashes. Fix this by doing what the comment in the post-process helper states, and just keep the whole output of "cargo vendor", by just removing the "tail --lines=+2". Since that comment is no longer meaningful, we drop it too. Now, all our 6 cargo-based packages, as well as our 5 python packages that have rust code, can be vendored again, without changing our hashes, but most importantly, with valid archives. Still, we keep the comment above the versions strings, in the hope that a future bumper will notice and be more careful at validating the vendoring. [0] https://github.com/rust-lang/cargo/pull/11273 Fixes: http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) ... and so many others... Reported-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: James Hilliard <james.hilliard1@gmail.com> Cc: Simon Richter <simon.richter@ptwdosimetry.com> Reviewed-by: James Hilliard <james.hilliard1@gmail.com> Reviewed-by: Romain Naour <romain.naour@smile.fr> Tested-by: Romain Naour <romain.naour@smile.fr> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-01-09 17:30:17 +01:00
| tee .cargo/config
popd > /dev/null
post_process_repack "$(pwd)" "${base_name}" "${output}"