kumquat-buildroot/support/download/svn

90 lines
3.0 KiB
Plaintext
Raw Normal View History

#!/usr/bin/env bash
# NOTE: if the output of this backend has to change (e.g. we change what gets
# included in the archive, or we change the format of the archive (e.g. tar
# options, compression ratio or method)), we MUST update the format version
# in the variable BR_FTM_VERSION_svn, in package/pkg-download.mk.
# We want to catch any unexpected failure, and exit immediately
set -e
# Download helper for svn, to be called from the download wrapper script
#
2018-04-02 10:14:22 +02:00
# Options:
# -q Be quiet.
# -o FILE Generate archive in FILE.
# -u URI Checkout from repository at URI.
# -c REV Use revision REV.
# -n NAME Use basename NAME.
# -r Recursive, i.e. use externals
#
# Environment:
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads DL_DIR can be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds. We do not want to trash this location with our temporary downloads (e.g. git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs). Turns out that we already have some kind of scratchpad, the BUILD_DIR. Although it is not really a disposable location, that's the best we have so far. Also, we create the temporary tarballs with mktemp using the final tarball, as template, since we want the temporary to be on the same filesystem as the final location, so the 'mv' is just a plain, atomic rename(2), and we are not left with a half-copied file as the final location. Using mktemp ensures all temp file names are unique, so it allows for parallel downloads from different build dirs at the same time, without cloberring each downloads. Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Samuel Martin <s.martin49@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> [tested a particular scenario that used to fail: two separate builds using a shared DL_DIR, ccache enabled, so that they run almost synchronously. These would download the same file at the same time, corrupting each other. With the patches in this series, all works fine.] Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-07-03 21:36:20 +02:00
# SVN : the svn command to call
# shellcheck disable=SC1090 # Only provides mk_tar_gz()
. "${0%/*}/helpers"
quiet=
externals=--ignore-externals
2018-04-02 10:14:22 +02:00
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) quiet=-q;;
2018-04-02 10:14:22 +02:00
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
c) rev="${OPTARG}";;
n) basename="${OPTARG}";;
r) externals=;;
2018-04-02 10:14:22 +02:00
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
2018-04-02 10:14:22 +02:00
shift $((OPTIND-1)) # Get rid of our options
support/download: protect from custom commands with spaces in args Some users may provide custom download commands with spaces in their arguments, like so: BR2_HG="hg --config foo.bar='some space-separated value'" However, the way we currently call those commands does not account for the extra quotes, and each space-separated part of the command is interpreted as separate arguments. Fix that by calling 'eval' on the commands. Because of the eval, we must further quote our own arguments, to avoid the eval further splitting them in case there are spaces (even though we do not support paths with spaces, better be clean from the onset to avoid breakage in the future). We change all the wrappers to use a wrapper-function, even those with a single call, so they all look alike. Note that we do not single-quote some of the variables, like ${verbose} because it can be empty and we really do not want to generate an empty-string argument. That's not a problem, as ${verbose} would not normally contain space-separated values (it could get set to something like '-q -v' but in that case we'd still want two arguments, so that's fine). Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-07 10:26:55 +01:00
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
_svn() {
if [ -z "${quiet}" ]; then
printf '%s ' "${SVN}" "${@}"; printf '\n'
fi
_plain_svn "$@"
}
# Note: please keep command below aligned with what is printed above
_plain_svn() {
# shellcheck disable=SC2086 # We want word-splitting for SVN
support/download: protect from custom commands with spaces in args Some users may provide custom download commands with spaces in their arguments, like so: BR2_HG="hg --config foo.bar='some space-separated value'" However, the way we currently call those commands does not account for the extra quotes, and each space-separated part of the command is interpreted as separate arguments. Fix that by calling 'eval' on the commands. Because of the eval, we must further quote our own arguments, to avoid the eval further splitting them in case there are spaces (even though we do not support paths with spaces, better be clean from the onset to avoid breakage in the future). We change all the wrappers to use a wrapper-function, even those with a single call, so they all look alike. Note that we do not single-quote some of the variables, like ${verbose} because it can be empty and we really do not want to generate an empty-string argument. That's not a problem, as ${verbose} would not normally contain space-separated values (it could get set to something like '-q -v' but in that case we'd still want two arguments, so that's fine). Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-07 10:26:55 +01:00
eval ${SVN} "${@}"
}
# shellcheck disable=SC2086 # externals and quiet may be empty
_svn export --ignore-keywords ${quiet} ${externals} "${@}" "'${uri}@${rev}'" "'${basename}'"
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads DL_DIR can be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds. We do not want to trash this location with our temporary downloads (e.g. git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs). Turns out that we already have some kind of scratchpad, the BUILD_DIR. Although it is not really a disposable location, that's the best we have so far. Also, we create the temporary tarballs with mktemp using the final tarball, as template, since we want the temporary to be on the same filesystem as the final location, so the 'mv' is just a plain, atomic rename(2), and we are not left with a half-copied file as the final location. Using mktemp ensures all temp file names are unique, so it allows for parallel downloads from different build dirs at the same time, without cloberring each downloads. Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Samuel Martin <s.martin49@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> [tested a particular scenario that used to fail: two separate builds using a shared DL_DIR, ccache enabled, so that they run almost synchronously. These would download the same file at the same time, corrupting each other. With the patches in this series, all works fine.] Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-07-03 21:36:20 +02:00
# For 'svn info', we only need the credentials, if any; other options
# would be invalid, as they are intended for 'svn export'.
# We can also consume the positional parameters, as we'll no longer
# be calling any other remote-reaching svn command.
creds=
while [ ${#} -gt 0 ]; do
case "${1}" in
--username=*) creds+=" ${1}"; shift;;
--password=*) creds+=" ${1}"; shift;;
--username) creds+=" ${1} ${2}"; shift 2;;
--password) creds+=" ${1} ${2}"; shift 2;;
*) shift;;
esac
done
# Get the date of the revision, to generate reproducible archives.
# The output format is YYYY-MM-DDTHH:MM:SS.mmmuuuZ (i.e. always in the
# UTC timezone), which we can feed as-is to the --mtime option for tar.
# In case there is a redirection (e.g. http -> https), just keep the
# last line (svn outputs everything on stdout)
# shellcheck disable=SC2086 # creds may be empty
date="$( _plain_svn info ${creds} "'${uri}@${rev}'" \
support/download: make the svn backend more reproducible Since c043ecb20ce6 (support/download: change format of archives generated from svn), the svn backend uses the generic helper to create reproducible archives. That helper really does its job as expected, but the svn backend is flawed in two ways: - the first, most obvious breakage happens with versions older than 1.9, as they do not support the '--show-item' option for the 'info' action; - the second is more involved, in that svn will by default expand the old, legacy, deprecated, cumbersome CVS-style keywords, in the form of revision marks like '$Date$' in a C-style comment in a source file. These replacements are done on checkout as well as on export, and they use local settings, like the local locale and timezone. This means that two people with different settings, will get different sources when the svn-checkout or svn-export the same revision from the same tree... Needless to say that this is not very reproducible... While the first is easily solved, the second is more involved. We need to ensure that what source is used initially to compute the hash, will also be the source that are used to check the hash. There are basically two solutions: 1. we ensure the same environment, by forcing the timezone and the locale to arbitrary values 2. we disable keyword expansion For the first solution, this still leaves the possibility that we miss some environment settings that have an impact on the keyword expansion. It would mean that Yann's settings be used, as he did introduce the hash for the only svn-downloaded package we have, avrdude, settings which are: TZ=Europe/Paris LC_TIME="en_US.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="fr_FR.utf8" LC_NUMERIC="fr_FR.utf8" The second option means that the generated archives change. That means we'd have to bump the archive version for svn downloads, and that we update the hashes for all the svn-downloaded packages. We chose to go with the second option, because this is what really makes more sense, rather than hard-coding arbitrary values in the environment. And we also have only one svn-downloaded package, avrdude. And thus, we're reaching the trigger for this change: avrdude is impacted by the CVS-keyword expansion issue: https://svn.savannah.gnu.org/viewvc/avrdude/trunk/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js?revision=1396&view=markup which would give two different files when checked out on different machines: diff -durN foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js --- foo/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200 +++ bar/avrdude-r1450/avrdude/atmel-docs/EDBG/common/jquery/layout/jquery.layout.js 2020-09-22 09:36:45.000000000 +0200 @@ -1,6 +1,6 @@ /** * @preserve jquery.layout 1.3.0 - Release Candidate 30.51 - * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $ + * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $ * $Rev: 303005 $ * * Copyright (c) 2012 @@ -4718,7 +4718,7 @@ /** * jquery.layout.state 1.0 - * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $ + * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $ * * Copyright (c) 2010 * Kevin Dalman (http://allpro.net) @@ -5074,7 +5074,7 @@ /** * jquery.layout.buttons 1.0 - * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $ + * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $ * * Copyright (c) 2010 * Kevin Dalman (http://allpro.net) @@ -5356,7 +5356,7 @@ /** * jquery.layout.browserZoom 1.0 - * $Date: 2015-11-02 22:13:28 +0100 (Mon, 02 Nov 2015) $ + * $Date: 2015-11-02 21:13:28 +0000 (Mon, 02 Nov 2015) $ * * Copyright (c) 2012 * Kevin Dalman (http://allpro.net) So we also update the hash for avrdude. Fixes: http://autobuild.buildroot.org/results/e3b/e3b0508047f32008ebfa83c5255ec5994b6af120/ (time issue) http://autobuild.buildroot.org/results/48e/48e78e84b425e79cdb98c16ab40247a0fa7e9676/ (keyword expansion issue) Reported-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Vincent Fazio <vfazio@xes-inc.com> Cc: Alexander Sverdlin <alexander.sverdlin@gmail.com> Reviewed-by: Alexander Sverdlin <alexander.sverdlin@gmail.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-03-10 23:59:37 +01:00
|sed -r -e '/^Last Changed Date: /!d; s///'
)"
# Generate the archive.
# We did a 'svn export' above, so it's not a working copy (there is no .svn
# directory or file to ignore).
mk_tar_gz "${basename}" "${basename}" "${date}" "${output}"