2014-10-21 16:05:56 +02:00
|
|
|
#!/usr/bin/env bash
|
2014-07-02 23:11:20 +02:00
|
|
|
|
2020-12-28 12:06:11 +01:00
|
|
|
# NOTE: if the output of this backend has to change (e.g. we change what gets
|
|
|
|
# included in the archive, or we change the format of the archive (e.g. tar
|
|
|
|
# options, compression ratio or method)), we MUST update the format version
|
|
|
|
# in the variable BR_FTM_VERSION_svn, in package/pkg-download.mk.
|
|
|
|
|
2014-08-03 19:53:41 +02:00
|
|
|
# We want to catch any unexpected failure, and exit immediately
|
2014-07-02 23:11:20 +02:00
|
|
|
set -e
|
|
|
|
|
2014-08-03 19:53:41 +02:00
|
|
|
# Download helper for svn, to be called from the download wrapper script
|
2015-01-02 16:53:39 +01:00
|
|
|
#
|
2018-04-02 10:14:22 +02:00
|
|
|
# Options:
|
|
|
|
# -q Be quiet.
|
|
|
|
# -o FILE Generate archive in FILE.
|
|
|
|
# -u URI Checkout from repository at URI.
|
|
|
|
# -c REV Use revision REV.
|
|
|
|
# -n NAME Use basename NAME.
|
2015-01-02 16:53:39 +01:00
|
|
|
#
|
|
|
|
# Environment:
|
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.
We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.
Also, we create the temporary tarballs with mktemp using the final tarball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.
Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-07-03 21:36:20 +02:00
|
|
|
# SVN : the svn command to call
|
2014-07-02 23:11:20 +02:00
|
|
|
|
2020-12-28 12:06:11 +01:00
|
|
|
. "${0%/*}/helpers"
|
|
|
|
|
2015-01-02 16:53:39 +01:00
|
|
|
verbose=
|
2018-04-02 10:14:22 +02:00
|
|
|
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
|
2015-01-02 16:53:39 +01:00
|
|
|
case "${OPT}" in
|
|
|
|
q) verbose=-q;;
|
2018-04-02 10:14:22 +02:00
|
|
|
o) output="${OPTARG}";;
|
|
|
|
u) uri="${OPTARG}";;
|
|
|
|
c) rev="${OPTARG}";;
|
|
|
|
n) basename="${OPTARG}";;
|
|
|
|
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
|
2015-01-02 16:53:39 +01:00
|
|
|
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
|
|
|
|
esac
|
|
|
|
done
|
|
|
|
|
2018-04-02 10:14:22 +02:00
|
|
|
shift $((OPTIND-1)) # Get rid of our options
|
2016-08-23 14:19:44 +02:00
|
|
|
|
support/download: protect from custom commands with spaces in args
Some users may provide custom download commands with spaces in their
arguments, like so:
BR2_HG="hg --config foo.bar='some space-separated value'"
However, the way we currently call those commands does not account
for the extra quotes, and each space-separated part of the command is
interpreted as separate arguments.
Fix that by calling 'eval' on the commands.
Because of the eval, we must further quote our own arguments, to avoid
the eval further splitting them in case there are spaces (even though
we do not support paths with spaces, better be clean from the onset to
avoid breakage in the future).
We change all the wrappers to use a wrapper-function, even those with
a single call, so they all look alike.
Note that we do not single-quote some of the variables, like ${verbose}
because it can be empty and we really do not want to generate an
empty-string argument. That's not a problem, as ${verbose} would not
normally contain space-separated values (it could get set to something
like '-q -v' but in that case we'd still want two arguments, so that's
fine).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-07 10:26:55 +01:00
|
|
|
# Caller needs to single-quote its arguments to prevent them from
|
|
|
|
# being expanded a second time (in case there are spaces in them)
|
|
|
|
_svn() {
|
|
|
|
eval ${SVN} "${@}"
|
|
|
|
}
|
|
|
|
|
2018-04-02 10:14:22 +02:00
|
|
|
_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
|
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.
We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.
Also, we create the temporary tarballs with mktemp using the final tarball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.
Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-07-03 21:36:20 +02:00
|
|
|
|
2019-12-22 22:31:49 +01:00
|
|
|
# Get the date of the revision, to generate reproducible archives.
|
|
|
|
# The output format is YYYY-MM-DDTHH:MM:SS.mmmuuuZ (i.e. always in the
|
|
|
|
# UTC timezone), which we can feed as-is to the --mtime option for tar.
|
|
|
|
# In case there is a redirection (e.g. http -> https), just keep the
|
|
|
|
# last line (svn outputs everything on stdout)
|
|
|
|
date="$( _svn info --show-item last-changed-date "'${uri}@${rev}'" |tail -n 1 )"
|
|
|
|
|
2020-12-28 12:06:11 +01:00
|
|
|
# Generate the archive.
|
2020-12-23 22:21:05 +01:00
|
|
|
# We did a 'svn export' above, so it's not a working copy (there is no .svn
|
|
|
|
# directory or file to ignore).
|
2020-12-28 12:06:11 +01:00
|
|
|
mk_tar_gz "${basename}" "${basename}" "${date}" "${output}"
|