From fbddd126011611e97cc920e40cd58784e29f242d Mon Sep 17 00:00:00 2001 From: "Yann E. MORIN" Date: Sat, 14 Jan 2023 15:48:53 +0100 Subject: [PATCH] support/download: fix concurrent cargo vendor Commit 8450b7691870 (package/pkg-cargo: move CARGO_HOME into DL_DIR) allowed for a shared cargo cache of crates. Internally, cargo is supposed to lock themselves when accessing that cache, and that commit even had some research in that area, pointing at [0] for complaints about too-coarse the lock, so it was deemed safe to have a shared cargo home. However, in practice, the locking as implemented by cargo, fails to properly protect the concurrent accesses to the crates cache, with random failures that manifest themselves like so: Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Downloading crates ... error: failed to sync Caused by: failed to download packages Caused by: failed to download `autocfg v1.1.0` Caused by: unable to get packages from source Caused by: failed to unpack package `autocfg v1.1.0` Caused by: failed to unpack entry at `autocfg-1.1.0/src/tests.rs` Caused by: No such file or directory (os error 2) while canonicalizing [...] with the last few errors sometime being: Caused by: failed to parse manifest at `[...]/aho-corasick-0.7.18/Cargo.toml` Caused by: can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path So, as we do not systematically use our own cargo build (we can use a pre-built one with host-rust-bin), we can't patch cargo (even if we knew what to do!). Instead, we implement a lock ourselves, by wrapping the call to "cargo vendor" with a flock(1) on cargo home. Note: the download wrapper is already flock-ed, but it is a per-package lock, so it does not prevent different packages from being downloaded in parallel; if those packages need cargo vendoring, that will not be protected by the flock on the dl wrapper. So we really do need a flock on cargo home. [0] https://github.com/rust-lang/cargo/issues/6930 Fixes: 8450b769187087751f83cbefcf0a88f70d9da670 Signed-off-by: Yann E. MORIN Cc: Thomas Petazzoni Cc: Moritz Bitsch Signed-off-by: Thomas Petazzoni --- support/download/cargo-post-process | 2 ++ 1 file changed, 2 insertions(+) diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process index 21a6be8dbe..aea2d8da7a 100755 --- a/support/download/cargo-post-process +++ b/support/download/cargo-post-process @@ -26,6 +26,8 @@ pushd "${base_name}" > /dev/null # Create the local .cargo/config with vendor info mkdir -p .cargo/ +mkdir -p "${CARGO_HOME}" +flock "${CARGO_HOME}"/.br-lock \ cargo vendor \ --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ --locked VENDOR \