2014-10-21 16:05:56 +02:00
|
|
|
#!/usr/bin/env bash
|
2014-07-02 23:11:19 +02:00
|
|
|
|
2014-08-03 19:53:38 +02:00
|
|
|
# We want to catch any unexpected failure, and exit immediately
|
2014-07-02 23:11:19 +02:00
|
|
|
set -e
|
|
|
|
|
2014-08-03 19:53:38 +02:00
|
|
|
# Download helper for git, to be called from the download wrapper script
|
2015-01-02 16:53:39 +01:00
|
|
|
#
|
|
|
|
# Call it as:
|
2016-07-01 11:01:18 +02:00
|
|
|
# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
|
|
|
|
#
|
|
|
|
# -q Be quiet.
|
|
|
|
# -r Clone and archive sub-modules.
|
2015-01-02 16:53:39 +01:00
|
|
|
#
|
|
|
|
# Environment:
|
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.
We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.
Also, we create the temporary tarballs with mktemp using the final tarball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.
Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-07-03 21:36:20 +02:00
|
|
|
# GIT : the git command to call
|
2014-07-02 23:11:19 +02:00
|
|
|
|
2015-07-26 12:26:26 +02:00
|
|
|
verbose=
|
2016-07-01 11:01:18 +02:00
|
|
|
recurse=0
|
|
|
|
while getopts :qr OPT; do
|
2015-01-02 16:53:39 +01:00
|
|
|
case "${OPT}" in
|
2015-01-04 19:10:14 +01:00
|
|
|
q) verbose=-q; exec >/dev/null;;
|
2016-07-01 11:01:18 +02:00
|
|
|
r) recurse=1;;
|
2015-01-02 16:53:39 +01:00
|
|
|
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
|
|
|
|
esac
|
|
|
|
done
|
|
|
|
shift $((OPTIND-1))
|
|
|
|
|
2014-08-03 19:53:38 +02:00
|
|
|
output="${1}"
|
|
|
|
repo="${2}"
|
|
|
|
cset="${3}"
|
|
|
|
basename="${4}"
|
2014-07-02 23:11:19 +02:00
|
|
|
|
2016-08-23 14:19:44 +02:00
|
|
|
shift 4 # Get rid of our options
|
|
|
|
|
support/download: protect from custom commands with spaces in args
Some users may provide custom download commands with spaces in their
arguments, like so:
BR2_HG="hg --config foo.bar='some space-separated value'"
However, the way we currently call those commands does not account
for the extra quotes, and each space-separated part of the command is
interpreted as separate arguments.
Fix that by calling 'eval' on the commands.
Because of the eval, we must further quote our own arguments, to avoid
the eval further splitting them in case there are spaces (even though
we do not support paths with spaces, better be clean from the onset to
avoid breakage in the future).
We change all the wrappers to use a wrapper-function, even those with
a single call, so they all look alike.
Note that we do not single-quote some of the variables, like ${verbose}
because it can be empty and we really do not want to generate an
empty-string argument. That's not a problem, as ${verbose} would not
normally contain space-separated values (it could get set to something
like '-q -v' but in that case we'd still want two arguments, so that's
fine).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-07 10:26:55 +01:00
|
|
|
# Caller needs to single-quote its arguments to prevent them from
|
|
|
|
# being expanded a second time (in case there are spaces in them)
|
|
|
|
_git() {
|
|
|
|
eval ${GIT} "${@}"
|
|
|
|
}
|
|
|
|
|
2015-05-28 13:35:28 +02:00
|
|
|
# Try a shallow clone, since it is faster than a full clone - but that only
|
|
|
|
# works if the version is a ref (tag or branch). Before trying to do a shallow
|
|
|
|
# clone we check if ${cset} is in the list provided by git ls-remote. If not
|
|
|
|
# we fall back on a full clone.
|
|
|
|
#
|
|
|
|
# Messages for the type of clone used are provided to ease debugging in case of
|
|
|
|
# problems
|
2014-08-06 18:47:52 +02:00
|
|
|
git_done=0
|
support/download: protect from custom commands with spaces in args
Some users may provide custom download commands with spaces in their
arguments, like so:
BR2_HG="hg --config foo.bar='some space-separated value'"
However, the way we currently call those commands does not account
for the extra quotes, and each space-separated part of the command is
interpreted as separate arguments.
Fix that by calling 'eval' on the commands.
Because of the eval, we must further quote our own arguments, to avoid
the eval further splitting them in case there are spaces (even though
we do not support paths with spaces, better be clean from the onset to
avoid breakage in the future).
We change all the wrappers to use a wrapper-function, even those with
a single call, so they all look alike.
Note that we do not single-quote some of the variables, like ${verbose}
because it can be empty and we really do not want to generate an
empty-string argument. That's not a problem, as ${verbose} would not
normally contain space-separated values (it could get set to something
like '-q -v' but in that case we'd still want two arguments, so that's
fine).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-12-07 10:26:55 +01:00
|
|
|
if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
|
2014-07-02 23:11:19 +02:00
|
|
|
printf "Doing shallow clone\n"
|
2016-08-23 14:19:44 +02:00
|
|
|
if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
|
2014-07-07 23:44:33 +02:00
|
|
|
git_done=1
|
2014-08-03 19:53:38 +02:00
|
|
|
else
|
|
|
|
printf "Shallow clone failed, falling back to doing a full clone\n"
|
2014-07-07 23:44:33 +02:00
|
|
|
fi
|
|
|
|
fi
|
|
|
|
if [ ${git_done} -eq 0 ]; then
|
2014-07-02 23:11:19 +02:00
|
|
|
printf "Doing full clone\n"
|
2016-08-23 14:19:44 +02:00
|
|
|
_git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
|
support/download/git: do not use bare clones
Currently, we are using bare clones, so as to minimise the disk usage,
most notably for largeish repositories such as the one for the Linux
kernel, which can go beyond the 1GiB barrier.
However, this precludes updating (and thus using) the submodules, if
any, of the repositories, as a working copy is required to use
submodules (becaue we need to know the list of submodules, where to find
them, where to clone them, what cset to checkout, and all those is
dependent upon the checked out cset of the father repository).
Switch to using /plain/ clones with a working copy.
This means that the extra refs used by some forges (like pull-requests
for Github, or changes for gerrit...) are no longer fetched as part of
the clone, because git does not offer to do a mirror clone when there is
a working copy.
Instead, we have to fetch those special refs by hand. Since there is no
easy solution to know whether the cset the user asked for is such a
special ref or not, we just try to always fetch the cset requested by
the user; if this fails, we assume that this is not a special ref (most
probably, it is a sha1) and we defer the check to the archive creation,
which would fail if the requested cset is missing anyway.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:16 +02:00
|
|
|
fi
|
|
|
|
|
|
|
|
pushd "${basename}" >/dev/null
|
|
|
|
|
|
|
|
# Try to get the special refs exposed by some forges (pull-requests for
|
|
|
|
# github, changes for gerrit...). There is no easy way to know whether
|
|
|
|
# the cset the user passed us is such a special ref or a tag or a sha1
|
|
|
|
# or whatever else. We'll eventually fail at checking out that cset,
|
|
|
|
# below, if there is an issue anyway. Since most of the cset we're gonna
|
|
|
|
# have to clone are not such special refs, consign the output to oblivion
|
|
|
|
# so as not to alarm unsuspecting users, but still trace it as a warning.
|
|
|
|
if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
|
|
|
|
printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
|
2014-07-02 23:11:19 +02:00
|
|
|
fi
|
|
|
|
|
2016-07-01 11:01:18 +02:00
|
|
|
# Checkout the required changeset, so that we can update the required
|
|
|
|
# submodules.
|
support/download/git: do not use git archive, handle it manually
We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would
have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the
generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:17 +02:00
|
|
|
_git checkout -q "'${cset}'"
|
|
|
|
|
|
|
|
# Get date of commit to generate a reproducible archive.
|
|
|
|
# %cD is RFC2822, so it's fully qualified, with TZ and all.
|
2016-07-28 10:40:22 +02:00
|
|
|
date="$( _git log -1 --pretty=format:%cD )"
|
support/download/git: do not use git archive, handle it manually
We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would
have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the
generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:17 +02:00
|
|
|
|
2016-07-01 11:01:18 +02:00
|
|
|
# There might be submodules, so fetch them.
|
|
|
|
if [ ${recurse} -eq 1 ]; then
|
|
|
|
_git submodule update --init --recursive
|
|
|
|
fi
|
|
|
|
|
2016-07-04 09:52:25 +02:00
|
|
|
# We do not need the .git dir; we keep other .git files, in case they
|
|
|
|
# are the only files in their directory.
|
|
|
|
rm -rf .git
|
2014-07-02 23:11:19 +02:00
|
|
|
|
support/download/git: do not use bare clones
Currently, we are using bare clones, so as to minimise the disk usage,
most notably for largeish repositories such as the one for the Linux
kernel, which can go beyond the 1GiB barrier.
However, this precludes updating (and thus using) the submodules, if
any, of the repositories, as a working copy is required to use
submodules (becaue we need to know the list of submodules, where to find
them, where to clone them, what cset to checkout, and all those is
dependent upon the checked out cset of the father repository).
Switch to using /plain/ clones with a working copy.
This means that the extra refs used by some forges (like pull-requests
for Github, or changes for gerrit...) are no longer fetched as part of
the clone, because git does not offer to do a mirror clone when there is
a working copy.
Instead, we have to fetch those special refs by hand. Since there is no
easy solution to know whether the cset the user asked for is such a
special ref or not, we just try to always fetch the cset requested by
the user; if this fails, we assume that this is not a special ref (most
probably, it is a sha1) and we defer the check to the archive creation,
which would fail if the requested cset is missing anyway.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:16 +02:00
|
|
|
popd >/dev/null
|
|
|
|
|
support/download/git: do not use git archive, handle it manually
We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would
have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the
generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:17 +02:00
|
|
|
# Generate the archive, sort with the C locale so that it is reproducible
|
support/download: make the git wrapper more robust
Currently, there are two failure paths in the wrapper:
- if the tar fails, then the error is ignored because it is on the
left-hand-side of a pipe;
- if the find fails, then the error is ignored because it is a
process substitution (and there is a pipe, too).
While the former could be fixed with "set -o pipefail", the latter can
not be fixed thusly and we must use an intemediate file for it.
So, fix both issues by using intermediate files, both to generate the
list of files to include in the archive, and generate the archive in a
temporary tarball.
Fixes the following build issue, where the find is failing for whatever
unknown reason:
http://autobuild.buildroot.net/results/20f/20fd76d2256eee81837f7e9bbaefbe79d7645ae9/
And this one, where the process substitution failed, also for an unknown
reason:
http://autobuild.buildroot.org/results/018/018971ea9227b386fe25d3c264c7e80b843a9f68/
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-08-01 00:35:55 +02:00
|
|
|
find "${basename}" -not -type d >"${basename}.list"
|
|
|
|
LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
|
support/download/git: do not use git archive, handle it manually
We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would
have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the
generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2016-07-01 11:01:17 +02:00
|
|
|
tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \
|
support/download: make the git wrapper more robust
Currently, there are two failure paths in the wrapper:
- if the tar fails, then the error is ignored because it is on the
left-hand-side of a pipe;
- if the find fails, then the error is ignored because it is a
process substitution (and there is a pipe, too).
While the former could be fixed with "set -o pipefail", the latter can
not be fixed thusly and we must use an intemediate file for it.
So, fix both issues by using intermediate files, both to generate the
list of files to include in the archive, and generate the archive in a
temporary tarball.
Fixes the following build issue, where the find is failing for whatever
unknown reason:
http://autobuild.buildroot.net/results/20f/20fd76d2256eee81837f7e9bbaefbe79d7645ae9/
And this one, where the process substitution failed, also for an unknown
reason:
http://autobuild.buildroot.org/results/018/018971ea9227b386fe25d3c264c7e80b843a9f68/
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-08-01 00:35:55 +02:00
|
|
|
-T "${basename}.list.sorted" >"${output}.tar"
|
|
|
|
gzip -n <"${output}.tar" >"${output}"
|