3a6d53c9f0
2 Commits
Author | SHA1 | Message | Date | |
---|---|---|---|---|
Vincent Fazio
|
b50eef1dfd |
support/download: drop sub-second precision in tarball creation
Some download backends, like svn, will provide timestamps with a sub-second precision, e.g. $ svn info --show-item last-changed-date [...] 2021-02-19T20:22:34.889717Z However, the PAX headers do not accept sub-second precision, leading to failure to download from subversion: tar: Time stamp is out of allowed range tar: Exiting with failure status due to previous errors make[1]: *** [package/pkg-generic.mk:148: [...]/build/subversion-1886712/.stamp_downloaded] Error 1 Fix that by massaging the timestamp to drop the sub-second part. We do that in the generic helper, rather than the svn backend, so that all callers to the generic helper benefit from this, as this is more an internal details of the tarball limitations, than of the backends themselves. Reported-by: Roosen Henri <Henri.Roosen@ginzinger.com> Signed-off-by: Vincent Fazio <vfazio@xes-inc.com> [yann.morin.1998@free.fr: - add Henri as reporter - move it out of the svn backend, and to the generic helper - reword the commit log accordingly - use an explicit time format rather than -Iseconds ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> |
||
Yann E. MORIN
|
cbe95b1a45 |
support/download: add helper to generate a reproducible archive
We currently need to generate reproducible archives in at least two locations: the git and svn download backends. We also know of some future potential use (e.g. the other download backends, like cvs, or in the upcoming download post-processors for vendoring, like cargo and go). However, we are currently limited to a narrow range of tar versions that we support, to create reproducible archives, because the gnu format we use has changed with tar 1.30. As a consequence, and as time advances, more and more distros are, or will eventually start, shipping with tar 1.30 or later, and thus we need to always build our on host-tar. Now, thanks to some grunt work by Vincent, we have a set of options that we can pass tar, to generate reproducible archives back from tar-1.27 and up through tar-1.32, the latest released version. However, those options are non-trivial, so we do not want to have to repeat those (and maintain them) in multiple locations. Introduce a helper that can generate a reproducible archive from an input directory. The --pax-option, to set specific PAX headers, does not accept RFC2822 timestamps which value are too away from some fixed point (set atcompile-time?): tar: Time stamp is out of allowed range However, the same timestamps passed as strict compliant ISO 8601 are accepted, so that's what we expect as a date format. Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Vincent Fazio <vfazio@xes-inc.com> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Reviewed-by: Vincent Fazio <vfazio@xes-inc.com> ---8<------8<------8<------8<--- # Here is a Makefile used to test all the versions of tar, with # different output formats and different sets of options: # Versions prior to 1.27 do not build on recent machines, because # 'gets()' got removed (rightfully so), so don't count them as # candidates. VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32 DATE = Thu 21 May 2020 06:44:11 PM CEST TARS = \ $(patsubst %,test_gnu_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS)) all: $(TARS) sha1sum $(^) .INTERMEDIATE: test_%.tar test_gnu_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=gnu \ -T list \ >$(@) test_posix_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ -T list \ >$(@) test_posix_paxoption_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ --pax-option='delete=atime,delete=ctime,delete=mtime' \ --pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \ -T list \ >$(@) list: .FORCE list: test (cd test && find . -not -type d ) |LC_ALL=C sort >$(@) LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng test: .FORCE test: rm -rf test mkdir -p test/bar echo foo >test/Foo echo bar >test/bar/Bar ln -s bar/Bar test/buz echo long >test/Very-$(LONG)-filename ln test/Very-$(LONG)-filename \ test/short .PRECIOUS: tar.% tar.%: tar-% cd $(<) && ./configure $(MAKE) -C $(<) install -m 0755 $(<)/src/tar $(@) .PRECIOUS: tar-% tar-%: tar-%.tar.gz tar xzf $(<) .PRECIOUS: tar-%.tar.gz tar-%.tar.gz: wget "https://ftp.gnu.org/gnu/tar/$(@)" .FORCE: clean: rm -rf tar-* tar.* test_* test list ---8<------8<------8<------8<--- |