support/download: add helper to generate a reproducible archive

support/download: add helper to generate a reproducible archive

We currently need to generate reproducible archives in at least two
locations: the git and svn download backends. We also know of some
future potential use (e.g. the other download backends, like cvs, or
in the upcoming download post-processors for vendoring, like cargo
and go).

However, we are currently limited to a narrow range of tar versions
that we support, to create reproducible archives, because the gnu
format we use has changed with tar 1.30.

As a consequence, and as time advances, more and more distros are,
or will eventually start, shipping with tar 1.30 or later, and thus
we need to always build our on host-tar.

Now, thanks to some grunt work by Vincent, we have a set of options
that we can pass tar, to generate reproducible archives back from
tar-1.27 and up through tar-1.32, the latest released version.

However, those options are non-trivial, so we do not want to have
to repeat those (and maintain them) in multiple locations.

Introduce a helper that can generate a reproducible archive from
an input directory.

The --pax-option, to set specific PAX headers, does not accept
RFC2822 timestamps which value are too away from some fixed point
(set atcompile-time?):
    tar: Time stamp is out of allowed range

However, the same timestamps passed as strict compliant ISO 8601 are
accepted, so that's what we expect as a date format.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
    ---8<------8<------8<------8<---
    # Here is a Makefile used to test all the versions of tar, with
    # different output formats and different sets of options:
    # Versions prior to 1.27 do not build on recent machines, because
    # 'gets()' got removed (rightfully so), so don't count them as
    # candidates.
    VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32
    DATE = Thu 21 May 2020 06:44:11 PM CEST

    TARS = \
    	$(patsubst %,test_gnu_%.tar,$(VERSIONS)) \
    	$(patsubst %,test_posix_%.tar,$(VERSIONS)) \
    	$(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS))

    all: $(TARS)
    	sha1sum $(^)

    .INTERMEDIATE: test_%.tar
    test_gnu_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=gnu \
    		-T list \
    	>$(@)
    test_posix_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=posix \
    		-T list \
    	>$(@)
    test_posix_paxoption_%.tar: tar.% list
    	./$(<) cf - -C test \
    		--transform="s#^\./#test-version/#" \
    		--numeric-owner --owner=0 --group=0 \
    		--mtime="$(DATE)" \
    		--format=posix \
    		--pax-option='delete=atime,delete=ctime,delete=mtime' \
    		--pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \
    		-T list \
    	>$(@)

    list: .FORCE
    list: test
    	(cd test && find . -not -type d ) |LC_ALL=C sort >$(@)

    LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng
    test: .FORCE
    test:
    	rm -rf test
    	mkdir -p test/bar
    	echo foo >test/Foo
    	echo bar >test/bar/Bar
    	ln -s bar/Bar test/buz
    	echo long >test/Very-$(LONG)-filename
    	ln test/Very-$(LONG)-filename \
    	   test/short

    .PRECIOUS: tar.%
    tar.%: tar-%
    	cd $(<) && ./configure
    	$(MAKE) -C $(<)
    	install -m 0755 $(<)/src/tar $(@)

    .PRECIOUS: tar-%
    tar-%: tar-%.tar.gz
    	tar xzf $(<)

    .PRECIOUS: tar-%.tar.gz
    tar-%.tar.gz:
    	wget "https://ftp.gnu.org/gnu/tar/$(@)"

    .FORCE:

    clean:
    	rm -rf tar-* tar.* test_* test list
    ---8<------8<------8<------8<---