The way we use it, gzip will store the current time in the header, which
leads to unreproducible archives.
Fix that by telling gzip to not store the name and date of the file it
compresses, with the -n option. Since it compresses its stdin, there was
already no filename stored; now there's even no date stored.
Note: gzip has had -n since at least 1.2.4, released in 1993, so
virtually every gzip out there nowadays has it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some users may provide custom download commands with spaces in their
arguments, like so:
BR2_HG="hg --config foo.bar='some space-separated value'"
However, the way we currently call those commands does not account
for the extra quotes, and each space-separated part of the command is
interpreted as separate arguments.
Fix that by calling 'eval' on the commands.
Because of the eval, we must further quote our own arguments, to avoid
the eval further splitting them in case there are spaces (even though
we do not support paths with spaces, better be clean from the onset to
avoid breakage in the future).
We change all the wrappers to use a wrapper-function, even those with
a single call, so they all look alike.
Note that we do not single-quote some of the variables, like ${verbose}
because it can be empty and we really do not want to generate an
empty-string argument. That's not a problem, as ${verbose} would not
normally contain space-separated values (it could get set to something
like '-q -v' but in that case we'd still want two arguments, so that's
fine).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
specify the SHA of a reference different than a branch or tag.
For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
Github stores the pull requests under refs/pull/xxx/head.
When cloning a repository with --bare, you don't fetch these references.
This patch uses --mirror for a full clone, in order to give the user
access to all references of the Git repository.
Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
In 50c8b7e (support/download: support -q in all download backends), the
backend were made to respect the quietness of the main Makefile, when -s
is poassed on the 'make' command line. In doing so, they were all made
to be verbose by default.
However, the verbosity of some of the tools, like scp, is very high, and
is in fact intended for debug purposes.
Drop being verbose by default, just use whatever each tool deems normal
output. Only respect the quietness requested by the user.
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Following commit 95a572282e (pkg-infra: move the git download helper to a
script, 2014-07-02), move the comment describing the shallow clone trickery as
well. Merge this comment with the existing helper comment that was added in
7e40a1103a (support/download: convert git to use the wrapper, 2014-08-03).
Rename $($(PKG)_DL_VERSION) to ${cset} to match the helper code context.
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We expresely call printf in the git helper, calls which were not
addresed in the previous silent-build patchset.
Just redirect stdout to oblivion when being silent.
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Tested-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add an option flag to all backends, as well as the check-hash script, so
as to silence download helpers when the user wants a silent build.
Additionaly, make the default be verbose.
Inspired by Fabio's patch on git/svn.
[Thomas: fix a typo "Environemnt" -> "Environment"
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Not all systems have /bin/bash (e.g. NixOS[1] doesn't). Buildroot
already uses /usr/bin/env shebangs for other interpreters (perl,
python), so why not bash?
This changes only the shebangs used by Buildroot itself; stuff installed
to the target system is left unchanged.
With this applied I can run Buildroot unmodified on NixOS.
[1]: http://nixos.org/
Signed-off-by: Bjørn Forsman <bjorn.forsman@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The git helper uses gzip to compress the intermediate tarball. But gzip
removes the source file, and create a new file named by appending .gz to
the original file name.
Thus, we end up with output.gz, while the download wrapper expects jsut
output, and thus believes the downlaod failed.
Fix that by storing the tar from git to a temporary file, then pipe this
file to gzip's stdin, and redirect gzip's stdout to the output file.
Reported-by: Graham Newton <gnewton@peavey-eu.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This drastically simplifies the git helper, as it no longer has to deal
with atomically saving the downloaded archive.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
(Tested by running 'make fmc-fsl-sdk-source')
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When switching the git helper over to a shell script, a special case was
not carried over: in case the remote has the required reference, we
attempt a shallow clone, using --depth 1. However, this is not supported
when the remote is accessed with the http protocol.
Therefore, the download fails.
What happened before the conversion to a shell script was that the helper
in the Makefile would fallback to doing a full-clone.
This is the case and behaviour that were lost in the conversion.
To avoid making the script too complex, we only attempt a full clone if
needed. And we decide that a full clone is needed by default; we decide
it is unnecessary if the remote has the needed reference *and* the
shallow clone was successful.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.
We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.
Also, we create the temporary tarballs with mktemp using the final tarball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.
Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The git download helper is getting a bit more complex. Fixing it in the
Makefile when it breaks (like the recent breakage with a non-existing
sha1-cset) proves to be challenging, to say the least.
Move it into a shell script in support/download/git, which will make
it much easier to read, maintain, fix and enhance in the future.
[Peter: redirect pushd/popd output to /dev/null]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
