Commit Graph

721 Commits

Author SHA1 Message Date
Matt Weber
aa2eb4bd64 support/testing: test_hardening fix flake8 whitespace
Resolves:
support/testing/tests/core/test_hardening.py:25:42: E231 missing whitespace after ','

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-20 17:27:28 +02:00
Thomas Petazzoni
b6b766527e support/config-fragments/autobuild: test the ARM AArch64 toolchain
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-20 17:12:27 +02:00
Mark Corbin
be43be070f support/config-fragments: add RISC-V 64-bit to autobuild configs
Add a minimal RISC-V 64-bit autobuild configuration for the
internal toolchain with glibc.

Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-20 14:17:23 +02:00
Matt Weber
963f824511 support/testing/tests/core: SSP & hardening flags
Catch the commonly used options of SSP, Relro, and fortify.
Using the package targets of busybox and lighttpd.  This
can easily be expanded to a larger list.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-20 13:25:27 +02:00
Thomas Petazzoni
22c2b27249 support/testing: fix TestATFMarvell test case
This test case uses a too old U-Boot version, which is affected by the
infamous libfdt header conflict issue. We update U-Boot and ATF to
what is used in the current version of
solidrun_macchiatobin_mainline_defconfig, for which the problem no
longer exists.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/107860312

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-14 19:38:32 +02:00
Thomas Petazzoni
2a63cea587 support/testing: fix TestATFAllwinner test case
This test case uses a too old U-Boot version, which is affected by the
infamous libfdt header conflict issue. Let's update to U-Boot 2017.11,
which is used by our current bananapi_m64_defconfig that was the
inspiration for this test case.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/107860310

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-14 19:38:20 +02:00
Ricardo Martincoski
1009590870 support/testing: add python-twisted tests
Use a minimal script to listen to a port and check using netstat.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
[Thomas: increase the delay after starting the Twisted server, as 5
seconds was not enough for Python 3.x configurations.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 17:28:06 +02:00
Ricardo Martincoski
35bcb54e1d support/testing: add python-incremental tests
Add a simple test case that imports the module and asserts a version
string for a fake package is generated.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 17:27:23 +02:00
Ricardo Martincoski
ddc0ebdb7c support/testing: add python-txtorcon tests
Add a simple test case that imports the module.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 17:27:15 +02:00
Ricardo Martincoski
f389f26691 support/testing: add python-txaio tests
Add a simple test case that imports the module to use with twisted in
Python 2 and with asyncio in Python 3.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 17:27:12 +02:00
Ricardo Martincoski
7e02070914 support/testing: add python-autobahn tests
Add a simple test case that imports the module.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-11 17:27:04 +02:00
Thomas Petazzoni
b2c24f584d package/pkg-generic.mk: increase precision of timestamps
Currently, the timestamps that we keep in build-time.log use a
second-level precision. However, as we are going to introduce a new
type of graph to draw the time line of a build, this precision is
going to be insufficient, as a number of steps are so short that they
are not even one second long, and generally the rounding to the second
gives a not so great looking graph.

Therefore, we add to the timestamps the nanoseconds using the %N date
specifier. A milli-second precision would have been sufficient, but %N
is all what date(1) provides at the sub-second level.

Since this is changing the format of the build-time.log file, this
commit adjusts the support/scripts/graph-build-time script
accordingly, to account for the floating point numbers that we have as
timestamps.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-10 21:30:51 +02:00
Matt Weber
3b657e264c support/scripts/pkg-stats: URL check using threads
Adds a pool of worker threads to accelerate connection testing.

~7.5MB and 2% CPU per thread on a Intel i5-3230M CPU @ 2.60GHz.

Runtime is ~3min in parallel vs ~15min.

CC: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-09 10:11:47 +02:00
Matt Weber
b4da3b2603 support/scripts/pkg-stats: URL checking support
- Adds support to check if a package has a URL and if that URL
  is valid by doing a header request.
- Reports this information as part of the generated html output

The URL data is currently gathered from the URL string provided
in the Kconfig help sections for each package.

This check helps ensure the URLs are valid and can be used
for other scripting purposes as the product's home site/URL.
CPE XML generation is an example of a case that could use this
product URL as part of an automated update generation script.

CC: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-10-09 10:05:46 +02:00
Petr Vorel
6eacea5ae0 support/kconfig: bump to kconfig from Linux 4.17-rc2
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-20 23:14:38 +02:00
Johan Oudinet
497f7134fc support/scripts/mkusers: preserve group members
When the function add_one_group is called on an existing group,
make sure the members of this group are not removed in the process of
deleting then re-adding the group.

Signed-off-by: Johan Oudinet <johan.oudinet@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: add curly braces when referencing ${members}, as suggested by
Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-12 21:22:25 +02:00
Yann E. MORIN
63a0593566 download/git: fix code-style
This file uses leading spaces, not TABs.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-10 22:44:26 +02:00
Romain Naour
f6db3df799 dependencies: host-make version check
The host make program is already checked by dependencies.sh but we
want to check the version number even if Buildroot is able to use
GNU make >= 3.81 but some packages may require a more recent version.

For example, since version 2.28 [1], glibc requires GNU Make >= 4.0.

For packages requiring make >= 4.0, the package makefile must use:
<PKG>_DEPENDENCIES = $(BR2_MAKE_HOST_DEPENDENCY) ...
<PKG>_MAKE = $(BR2_MAKE)

BR2_MAKE1 is also available if needed.

[1] https://www.sourceware.org/ml/libc-alpha/2018-08/msg00003.html

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
[Thomas: remove extraction of "bugfix" part of the version, since it's
not used anywhere.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-09-08 23:36:36 +02:00
Peter Korsgaard
721e4cbb52 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-09-07 13:13:17 +02:00
Peter Korsgaard
339d550e92 Update for 2018.08
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-09-06 22:11:06 +02:00
Hollis Blanchard
38de434123 download: fix file:// BR2_PRIMARY_SITE (download cache)
wget is the only downloader currently usable with BR2_PRIMARY_SITE, and that
doesn't work at all for file:// URLs. The symptoms are these:

	support/download/dl-wrapper -c '2.4.47' -d '/PATH/build/sw/source/attr' -D '/PATH/build/sw/source' -f 'attr-2.4.47.src.tar.gz' -H 'package/attr//attr.hash' -n 'attr-2.4.47' -N 'attr' -o '/PATH/build/sw/source/attr/attr-2.4.47.src.tar.gz'  -u file\|urlencode+file:///NFS/buildroot_dl_cache/attr -u file\|urlencode+file:///NFS/buildroot_dl_cache -u http+http://download.savannah.gnu.org/releases/attr -u http\|urlencode+http://sources.buildroot.net/attr -u http\|urlencode+http://sources.buildroot.net  --
	file:///NFS/buildroot_dl_cache/attr/attr-2.4.47.src.tar.gz: Unsupported scheme `file'.
	ERROR: attr-2.4.47.src.tar.gz has wrong sha256 hash:
	ERROR: expected: 25772f653ac5b2e3ceeb89df50e4688891e21f723c460636548971652af0a859
	ERROR: got     : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
	ERROR: Incomplete download, or man-in-the-middle (MITM) attack

In the case of custom Linux kernel versions, this is fatal, because there isn't
necessarily a hash file to indicate that wget's empty tarball is wrong.

This seems to have been broken by commit c8ef0c03b0, because:
1. BR2_PRIMARY_SITE always appends "urlencode" (package/pkg-download.mk)
2. Anything with the "|urlencode" suffix in $uri will end up using wget due to
   the backend case wildcarding.
3. The wget backend rejects file:/// URLs ("unsupported scheme"), and we end up
   with an empty .tar.gz file in the downloads directory.

Fix that by shell-extracting the backend name from the left of "|". I'm not
positive if all URLs will have a "|", so this code only looks for a "|" left of
the "+".

Signed-off-by: Hollis Blanchard <hollis_blanchard@mentor.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-21 00:24:29 +02:00
Yann E. MORIN
3ab56f8a63 support/dependencies: check for system-provided bison and flex
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-20 16:01:42 +02:00
Angelo Compagnucci
8511a66fc1 support/misc: updating Vagrantfile to ubuntu 18.04
This patch updates the vagrant box to ubuntu bionic 64 and switches back
to the official ubuntu image cause the issues with the official image
are now solved.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-13 11:45:48 +02:00
Matt Weber
b25c39ebd9 support/testing/infra: rm extra log flush in builder.py
While integrating proxy support in builder.py, a log flush
was left in the code.  This commit cleans/removes that code.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-11 12:38:45 +02:00
Matt Weber
361bc21c52 support/testing: runtest proxy support
Allow builder.py to inherit the system proxy settings from
the env if they are present.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-10 22:36:10 +02:00
Ricardo Martincoski
5f201de0dc support/testing: add test for file capabilities
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-08-10 08:25:51 +02:00
Yann E. MORIN
4dc6c43703 support/download: remove help from wrapper
The download wrapper is a purely internal helper, and is not supposed to
be callable manually. No need to offer some help.

Besides, the help text was way out-dated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-09 23:47:54 +02:00
Damien Thébault
d8c0faa980 support/misc/toolchainfile.cmake.in: use TARGET_LDFLAGS for shared and module libraries
With cmake packages, we are only using TARGET_LDFLAGS for executables
and not for shared libraries.

This patch adds CMAKE_SHARED_LINKER_FLAGS and
CMAKE_MODULE_LINKER_FLAGS to the cmake toolchain file so that
buildroot TARGET_LDFLAGS are used for shared and module libraries.

Signed-off-by: Damien Thébault <damien.thebault@vitec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-07-19 09:33:37 +02:00
David De Grave (Essensium/Mind)
d0505b849f support/dependencies/dependencies.sh: add details for Redhat/Fedora
Add the i686 package list to install when using pre-built 32 bits
binaries with a redhat/fedora host distribution (glibc.i686 and
zlib.i686).

Signed-off-by: David De Grave (Essensium/Mind) <david.degrave@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-07-02 22:43:56 +02:00
Thomas Petazzoni
edca2cd086 support/config-fragments/autobuild: update pre-built Buildroot toolchains
All pre-built Buildroot toolchains have been rebuilt with Buildroot
2018.05, so this commit updates the corresponding configuration
fragments to make sure the autobuilders use the new toolchains.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-30 18:00:30 +02:00
Ofer Heifetz
78968330e3 support/config-fragments/autobuild: add Linaro AArch64 BE support
Signed-off-by: Ofer Heifetz <oferh@marvell.com>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-28 22:15:57 +02:00
Yann E. MORIN
567a8476ad support/docker: use the distro-provided flake8
Currently, we install flake8 and its dependencies via pip. We
tried to be reproducible by pinning the version of those python
packages, but we did forget quite a few of them, and thus some
dependencies for flake8 are installed as uncontrolled versions.

Furthermore, before we install flake8 and its dependencies, we
forcibly update pip, setuptools, and wheels packages to their
latest versions. This explicitly breaks reproducibility.

While we could enforce a specific version of all those packages
and still grab them from PyPI, we can simply grab them from the
distribution-provided packages instead.

Since we're using a pinned version of stretch, this already
guarantees we'll reproducibly get the same versions over and
over again. Besides, we just need to list flake8 as a package to
install to automatically get all its dependencies (again, in a
reproducible way).

This has the slight unfortunate drawback of downgrading flake8
to version 3.2.1, from version 3.5.0, as well as downgrading a
few of flake8's dependencies, as noticed by Ricardo:
    http://lists.busybox.net/pipermail/buildroot/2018-May/222376.html

However, as Ricardo said, there isn't "any serious limitation of
this old version, the release notes for a version in the between
mentions 'Dramatically improve the performance' but we have a
limited number of scripts and running on Gitlab for all of them
still takes less than 5 minutes".

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-17 20:54:02 +02:00
Yann E. MORIN
afc1ec7073 support/docker: sort the list of installed packages
As suggested in the docker best practices [0], order the package list
alphabetically, and list only one package per line.

This will be much usefull later, we need to update the list of installed
packages, like adding new ones for example.

[0] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#sort-multi-line-arguments

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-17 20:53:53 +02:00
Yann E. MORIN
99723554c0 support/testing: fix python syntax
Fix three issues with code style in our test infra:
  - 'print' is now a function,
  - exceptions need to be caught-assigned with the 'as' keyword,
  - old-style "%s"%() formatting is deprecated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: drop indices in format strings.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-10 15:56:25 +02:00
Fabrice Fontaine
dc5553d2ce dependencies.sh: add perl autodie for whois
Add dependency on perl autodie if whois package is selected

Fixes:
 - http://autobuild.buildroot.net/results/4d2ca14580cbc73f94e86ee834fac6d2541298ec

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-05 22:20:18 +02:00
Robert Beckett
a5ba72946e dl-wrapper: Fix support for URIs containing '+'
'+' is a valid character in a url. The current dl-wrapper gets the
URI scheme by dropping everything after the last '+' character, with
the intension of finding 'git' from e.g. 'git+https://uri'.

If a uri has a '+' anywhere in it, it ends up using too much of the
string as a scheme, and fails to match the handler properly.

An example of where this form of URI is used is when using deploy tokens
in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
where username for deploy token is of the form 'gitlab+deploy-token-<number>'.

Use the %% operator to search backwards until the last '+' character when
dropping the rest of the string as we know that the first '+'
in the string should be the scheme.

Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-04 21:59:55 +02:00
Yann E. MORIN
c4f844ff32 support/docker: run apt-get update and apt-get install in two RUNs
In commit 7517aef4d (support/docker: limit the number of layers),
we reduced the number of layers by coalescing multiple RUN commands
into less commands.

In doing so, we especially coalesced "apt-get update" with "apt-get
install".

However, the distribution we used is a pinned version of stretch, so
we know that running apt-get update will always yield the same apt
database.

If we split the two apt-get commands, then we can re-use any local
intermediate image when we need to update the list of packages to
install; this helps quite a bit when testing the docker files over
and over again, with just slight variants in the packages list.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-06-03 20:31:20 +02:00
Peter Korsgaard
8b0fd3cb49 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-06-02 11:21:20 +02:00
Peter Korsgaard
f3d114a1ef Update for 2018.05
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-06-01 22:22:57 +02:00
Arnout Vandecappelle (Essensium/Mind)
66f40a7242 download/cvs: add a 10 minute timeout
Apparently, CVS servers can be deadlocked and in that case clients will
retry connecting to them indefinitely. Cfr.
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78/build-end.log
Apparently, the sf.net CVS server got in such a deadlock on 2018-05-18,
and almost 2 weeks later it is still not fixed.

Instead of just hanging, we should fall back on BR2_SECONDARY_SITE. To
achieve this, it's sufficient to add a timeout to the CVS command.

The timeout value is of course arbitrary. However, we can assume that
nobody will be putting large projects under CVS any more. So if the
download takes more than 5 minutes, it's probably broken. Let's put the
timeout at 10 minutes then.

Fixes:
http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105
http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78
http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-05-31 19:32:45 +02:00
Matt Weber
eb04235783 support/scripts/pkg-stats: use parenthesis for print
Use Python 3 style print calls, in order to make pkg-stats Python 3
compliant.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-19 13:48:21 +02:00
Angelo Compagnucci
43ca1987e6 support/download/file: remove set -x
Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-13 22:26:43 +02:00
Angelo Compagnucci
a9bb361435 support/download/file: fix file:// protocol handling
Since the rework of the download infrastructure, the "file" download
helper gets passed an URL that starts with file://, but forgets to
strip it before passing it to "cp", causing a failure as the "cp"
program isn't prepared for file paths starting with file://. This is
fixed by stripping the file:// at the beginning of the URL.

In addition, the path passed to cp lacked a slash between the
directory path and the filename part of the url. This is fixed by
adding a slash at the appropriate places.

Fixes the following build failure when the "file" download method is
used:

cp: cannot stat 'file:///home/angelo/DEV/TOOLCHAINSarmv7-eabihf--glibc--bleeding-edge-2017.11-1.tar.bz2': No such file or directory

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-13 22:23:13 +02:00
Petr Vorel
04918ca72f support/kconfig: Add missing target to README.buildroot
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-13 22:09:01 +02:00
Yann E. MORIN
72703d02b9 support/dependencies: check that PATH does not contain CWD
A person on IRC reported a build failure with the util-linux package,
looking like this:

for I in uname26 linux32 linux64        ; do \
	cd /home/aep/consulting/chargery/tracker/output/target/usr/bin && ln -sf setarch $I ; \
done
[...]
/bin/sh: line 1: ./ln: cannot execute binary file: Exec format error
/bin/sh: line 1: ./ln: cannot execute binary file: Exec format error
/bin/sh: line 1: ./ln: cannot execute binary file: Exec format error

The issue was an empty path in the PATH variable, which means "current
working directory", causing a "ln" binary built by util-linux for the
target to be used instead of the system-provided "ln".

We already check a number of things in the PATH and LD_LIBRARY_PATH
variables in support/dependencies/dependencies.sh, but we were not
checking that PATH did not contain an empty path.

This commit fixes that and takes this opportunity to simplify the test
code for PATH and LD_LIBRARY_PATH.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Thomas: improve commit log.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-09 23:26:18 +02:00
Yann E. MORIN
4f54c959dc download/git: always do full-clone
We currently attempt a shallow clone, as tentative to save bandwidth and
download time.

However, now that we keep the git tree as a cache, it may happen that we
need to checkout an earlier commit, and that would not be present with a
shallow clone.

Furthermore, the shallow fetch is already really broken, and just
happens to work by chance. Consider the following actions, which are
basically what happens today:

    mkdir git
    git init git
    cd git
    git remote add origin https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    git fetch origin --depth 1 v4.17-rc1
    if ! git fetch origin v4.17-rc1:v4.17-rc1 ; then
        echo "warning"
    fi
    git checkout v4.17-rc1

The checkout succeeds just because of the git-fetch in the if-condition,
which is initially there to fetch the special refs from github PRs, or
gerrit reviews. That fails, but we just print a warning. If we were to
ever remove support for special refs, then the checkout would fail.

The whole purpose of the git cache is to actually save bandwidth and
download time, but in the long run. For one-offs, people would
preferably use a wget download (e.g. with the github macro) instead of
a git clone.

We switch to always doing a full clone. It is more correct, and pays off
in the long run...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-01 21:22:28 +02:00
Yann E. MORIN
80d8bc6e46 download/git: ensure we can checkout repos with submodule conversions
When a git tree has had sub-dir <-> sub-module conversions, or has had
submodules added or removed over the course of time, checking out a
changeset across those conversions/additions/removals may leave
untracked files, or may fail because of a conflict of type.

So, before we checkout the new changeset, we forcibly remove the
submodules. The new set of submodules, if any, will be restored later.

Ideally, we would use a native git command: git submodule deinit --all.
However, that was only introduced in git 1.8.3 which, while not being
recent by modern standards, is still too old for some enterprise-grade
distributions (RHEL6 only has git-1.7.1).

So, instead, we just use git submodule foreach, to rm -rf the submodules
directory.

Again, we would ideally use 'cd $toplevel && rm -rf $path', but
$toplevel was only introduced in git 1.7.2. $path has always been there.

So, instead, we just cd back one level, and remove the basename of the
directory.

Eventually, we need to get rid of now-empty and untracked directories,
that were parents of a removed submodule. For example. ./foo/bar/ was a
submodule, so ./foo/bar/ was removed, which left ./foo/ around.

Yet again, recent-ish git versions would have removed it during the
forced checkout, but old-ish versions (e.g. 1.7.1) do not remove it with
the forced checkout.

Instead we rely on the already used forced-forced clean of directories,
untracked, and ignored content, to really get rid of extra stuff we are
not interested in.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-01 21:22:28 +02:00
Yann E. MORIN
3d2e018889 download/git: ensure we checkout to a clean state
Force the checkout to ignore and throw away any local changes. This
allows recovering from a previous partial checkout (e.g. killed by
the user, or by a CI job...)

git checkout -f has been supported since the inception of git, so we
can use it without any second thought.

Also do a forced-forced clean, to really get rid of all untracked stuff.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-01 21:22:28 +02:00
Yann E. MORIN
b7efb43e86 download/git: try to recover from utterly-broken repositories
In some cases, the repository may be in a state we can't automatically
recover from, especially since we must still support oldish git versions
that do not provide the necessary commands or options thereof.

As a last-ditch recovery, delete the repository and recreate the cache
from scratch.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-01 21:22:28 +02:00
Yann E. MORIN
428a0649ff download/git: quickly exit when the cset does not exist
Check that the given cset is indeed something we can checkout. If not,
then exit early.

This will be useful when a later commit will trap any failing git
command to try to recover the repository by doing a clone from scratch:
when the cset is not a commit, it does not mean the repository is broken
or what, and re-cloning from scratch would not help, so no need to trash
a good cache.

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-05-01 21:22:28 +02:00