Commit Graph

30 Commits

Author SHA1 Message Date
Yann E. MORIN
a07d37d226 toolchain/toolchain-wrapper.c: extend the list of unsafe paths
On some legacy systems, the X11 headers and libs are in /usr/X11R66/include
and /usr/X11R66/lib, and of course, some packages are trying to be smart
and use those paths (even when they do not exist).

Add those to the list of unsafe paths to check in the toolchain wrapper.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-22 22:28:08 +02:00
Peter Korsgaard
5fd8dd203a toolchain: use consistent code style for C code
Most, but not all our C code follows the Linux kernel code style (as
documented in Documentation/process/coding-style.rst).  Adjust the few
places doing differently:

- Braces:
  ..but the preferred way, as shown to us by the prophets Kernighan
  and Ritchie, is to put the opening brace last on the line

- Spaces after keywords:
  Use a space after (most) keywords

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-08 22:10:06 +01:00
Yann E. MORIN
7ba47d1cca toolchain/wrapper: also dump args it was called with
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-18 00:19:57 +02:00
Yann E. MORIN
814f6e19e7 toolchain: allow PIC/PIE without RELRO
In commit 7484c1c3b8 (toolchain/toolchain-wrapper: add BR2_RELRO_),
we added the PIC/PIE flags, but based on the RELRO_FULL condition.

It is however totally possible to do a PIC/PIE executable without
RELRO_FULL, as it is also valid to do a PIC/PIE build with RELRO_PARTIAL.

Add a new option that now governs the PIC/PIE flags.

Note: it is unknown if RELRO_FULL really needs PIC/PIE or not, so we
keep the current situation, where RELRO-FULL forces PIC/PIE compilation.
Decoupling can come later from an interested party.

Signed-off-by: "Yann E. MORIN" <yann.morin@orange.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 23:19:36 +02:00
Yann E. MORIN
b8ec113eb1 toolchain: set the ssp gcc option in kconfig
Currently, we repeat all the SSP level selection deep down to the
toolchain wrapper itself, where we eventually translate it to the
actual SSP option to use. This is a bit redundant.

Additionally, we will want to check that the toolchain actually
supports that option (for those toolchain where it was backported).

So, move the translation into kconfig, and add the qstrip'ed value
to the additional flags passed to the wrapper. Add it before
user-supplied opitons, to keep the previous behaviour (and allow
anyone crazy-enough to override it with BR2_TARGET_OPTIMIZATION).

Signed-off-by: "Yann E. MORIN" <yann.morin@orange.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-03-13 00:01:55 +01:00
Matt Weber
f10822d151 toolchain/toolchain-wrapper: add BR2_SSP_* support
Migrate the stack protection flag management into the wrapper.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-20 12:51:21 +02:00
Matt Weber
7484c1c3b8 toolchain/toolchain-wrapper: add BR2_RELRO_
The RELRO/PIE flags are currently passed via CFLAGS/LDFLAGS and this patch
proposes moving them to the toolchain wrapper.

 (1) The flags should _always_ be passed, without leaving the possibility
     for any package to ignore them. I.e, when BR2_RELRO_FULL=y is used
     in a build, all executables should be built PIE. Passing those
     options through the wrapper ensures they are used during the build
     of all packages.

 (2) Some options are incompatible with -fPIE. For example, when
     building object files for a shared libraries, -fPIC is used, and
     -fPIE shouldn't be used in combination with -fPIE. Similarly, -r
     or -static are directly incompatible as they are different link
     time behaviors then the intent of PIE. Passing those options
     through the wrapper allows to add some "smart" logic to only pass
     -fPIE/-pie when relevant.

 (3) Some toolchain, kernel and bootloader packages may want to
     explicitly disable PIE in a build where the rest of the userspace
     has intentionally enabled it. The wrapper provides an option
     to key on the -fno-pie/-no-pie and bypass the appending of RELRO
     flags.
     The current Kernel and U-boot source trees include this option.
     8438ee76b0
     6ace36e19a
     If using PIE with a older Kernel and/or U-boot version, a backport of these
     changes  might be required. However this patchset also uses the
     __KERNEL__ and __UBOOT__ defines as a way to disable PIE.

NOTE: The current implementation via CFLAGS/LDFLAGS has caused some
build time failures as the conditional logic doesn't yet exist in
Buildroot:

https://bugs.busybox.net/show_bug.cgi?id=11206
https://bugs.busybox.net/show_bug.cgi?id=11321

Good summary of the most common build failures related to
enabling pie: https://wiki.ubuntu.com/SecurityTeam/PIE

[Peter: minor cleanups]
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-20 12:50:29 +02:00
Waldemar Brodkorb
5e58509bfe toolchain-wrapper: use -ffp-contract=off on MIPS Xburst for gcc >= 4.6
Since gcc 4.6, GCC deprecated -mfused-madd, -ffp-contract=off should
be used for the Xburst workaround.

Tested with the MIPS Sourcery 2011.03 toolchain (based on gcc 4.5),
the toolchain wrapper uses -mno-fused-madd, as expected:

$ BR2_DEBUG_WRAPPER=2 ./output/host/bin/mips-linux-gnu-gcc -o toto toto.c
Toolchain wrapper executing:
    '/home/thomas/toolchains/mips-2011.03/bin/mips-linux-gnu-gcc'
    '--sysroot'
    '/home/thomas/projets/buildroot/output/host/mipsel-buildroot-linux-gnu/sysroot'
    '-mabi=32'
    '-msoft-float'
    '-mno-fused-madd'
    '-EL'
    '-march=mips32r2'
    '-o'
    'toto'
    'toto.c'

And with the MIPS Sourcery 2012.09 toolchain (based on gcc 4.7), the
toolchain wrapper uses -ffp-contract=off, as expected:

$ BR2_DEBUG_WRAPPER=2 ./output/host/bin/mips-linux-gnu-gcc -o toto toto.c
Toolchain wrapper executing:
    '/home/thomas/toolchains/mips-2012.09/bin/mips-linux-gnu-gcc'
    '--sysroot'
    '/home/thomas/projets/buildroot/output/host/mipsel-buildroot-linux-gnu/sysroot'
    '-mabi=32'
    '-msoft-float'
    '-ffp-contract=off'
    '-EL'
    '-march=mips32r2'
    '-o'
    'toto'
    'toto.c'

Fixes the ci20_defconfig build:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/60303132

Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
[Thomas: rework to continue supporting pre-gcc-4.6 toolchains, extend
the commit log after doing more testing.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 16:23:33 +02:00
Yann E. MORIN
6393b6904b toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc
Starting with version 7, gcc automatically recognises and enforces the
environment variable SOURCE_DATE_EPOCH, and fakes __DATE__ and __TIME__
accordingly, to produce reproducible builds (at least in regards to date
and time).

However, older gcc versions do not offer this feature.

So, we use our toolchain wrapper to force-feed __DATE__ and __TIME__ as
macros, which will take precedence over those that gcc may compute
itself. We compute them according to the specs:
    https://reproducible-builds.org/specs/source-date-epoch/
    https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html

Since we define macros otherwise internal to gcc, we have to tell it not
to warn about that. The -Wno-builtin-macro-redefined flag was introduced
in gcc-4.4.0. Therefore, we make BR2_REPRODUCIBLE depend on GCC >= 4.4.

gcc-7 will ignore SOURCE_DATE_EPOCH when __DATE__ and __TIME__ are
user-defined. Anyway, this is of no consequence: whether __DATE__ and
__TIME__ or SOURCE_DATE_EPOCH takes precedence, it would yield the
exact same end result since we use the same logic to compute it. Note
that we didn't copy the code for it from gcc so using the same logic
doesn't imply that we're inheriting GPL-3.0.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Jérôme Pouiller <jezz@sysmic.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
[Arnout: rewrite commit message]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-10-22 16:10:37 +02:00
Vicente Olivert Riera
9a0a0a976b arch/mips: add support for MIPS32 FP mode
MIPS32 support different FP modes (32,xx,64), so give the user the
opportunity to choose between them. That will cause host-gcc to be built
using the --with-fp-32=[32|xx|64] configure option. Also the
-mfp[32|xx|64] gcc option will be added to TARGET_CFLAGS and to the
toolchain wrapper.

FP mode option shouldn't be used for soft-float, so we add logic in the
toolchain wrapper if -msoft-float is among the arguments in order to not
append the -fp[[32|xx|64] option, otherwise the compilation may fail.

Information about FP modes here:

- https://sourceware.org/binutils/docs/as/MIPS-Options.html
- https://dmz-portal.imgtec.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking#5._Generating_modeless_code

Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-16 16:45:22 +02:00
Vicente Olivert Riera
2d8f3fc430 arch/mips: add support for MIPS NaN
MIPS supports two different NaN encodings, legacy and 2008. Information
about MIPS NaN encodings can be found here:

  https://sourceware.org/binutils/docs/as/MIPS-NaN-Encodings.html

NaN legacy is the only option available for R2 cores and older.
NaN 2008 is the only option available for R6 cores.
R5 cores can have either NaN legacy or NaN 2008, depending on the
implementation. So, if the user selects a generic R5 target architecture
variant, we show a choice menu with both options available. For well
known R5 cores we directly select the NaN enconding they use.

Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-16 16:35:39 +02:00
Arnout Vandecappelle
2a47bd37b5 toolchain-wrapper: remove remaining references to HOST_DIR/usr
Commit 14151d77af that eliminated
$(HOST_DIR)/usr seriously missed the toolchain-wrapper - only a single
reference was updated, the other three were missed. Commit
015d68c84c removed one more. This commit
finally removes the two remaining ones.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-10 17:46:11 +02:00
Arnout Vandecappelle
015d68c84c toolchain-wrapper: fix breakage after host/usr removal
The toolchain wrapper, when called through PATH, strips the last three
levels of /proc/self/exe to find HOST_DIR. However, after the host/usr
removal, this should be just two levels.

The toolchain wrapper has different logic for when it is called with a
full path (i.e. $HOST_DIR/usr/bin/arm-linux-gcc) then when it is called
through the PATH (i.e. just arm-linux-gcc). The latter is never used
internally in Buildroot, that's why this wasn't discovered through
testing.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Mark Jackson <mpfj-list@newflow.co.uk>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-07 09:57:14 +02:00
Arnout Vandecappelle
14151d77af Eliminate $(HOST_DIR)/usr
We currently use $(HOST_DIR)/usr as the prefix for host packages. That
has a few disadvantages:

- There are some things installed in $(HOST_DIR)/etc and
  $(HOST_DIR)/sbin, which is inconsistent.

- To pack a buildroot-built toolchain into a tarball for use as an
  external toolchain, you have to pack output/host/usr instead of the
  more obvious output/host.

- Because of the above, the internal toolchain wrapper breaks which
  forces us to work around it (call the actual toolchain executable
  directly). This is OK for us, but when used in another build system,
  that's a problem.

- Paths are four characters longer.

To allow us to gradually eliminate $(HOST_DIR)/usr while building
packages, replace it with a symlink to .

The symlinks from $(HOST_DIR)/usr/$(GNU_TARGET_NAME) and
$(HOST_DIR)/usr/lib that were added previously are removed again.

Note that the symlink creation will break when $(HOST_DIR)/usr
already exists as a directory, i.e. when rebuilding in an existing
output directory. This is necessary: if we don't break it now, the
following commits (which remove the usr part from various variables)
_will_ break it.

At the same time as creating this symlink, we have to update the
external toolchain wrapper and the external toolchain symlinks to go
one directory less up. Indeed, $(HOST_DIR) is one level less up than
it was before.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-07-05 11:45:35 +02:00
Yann E. MORIN
1c8c0d91a4 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
This reverts commit 76838f6341.

The commit referenced above explicitly states that the function was
copied as-is from the gcc source code at the time. And indeed that is
exactly how the function appeared in gcc in commit
e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.

However, our toolchain wrapper is "GPLv2 only", while the file this function
was copied from is "GPLv3 or later".  As such we can't include that function
and still comply to both licenses.

Furthermore, the code is far from optimum.

Since this feature is not release-critical, revert it until we re-implement
it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Jérôme Pouiller <jezz@sysmic.org>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-02-20 23:26:05 +01:00
Jérôme Pouiller
76838f6341 reproducible: fix DATE/TIME macros in toolchain-wrapper
The use of the __DATE__ and __TIME__ macros are one of the most common
sources of non-reproducible binaries. In order to fix that, gcc 7 supports
the SOURCE_DATE_EPOCH variable:

https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934

This patch take advantage of toolchain-wrapper to provide support of
SOURCE_DATE_EPOCH to older gcc versions.

Function get_source_date_epoch() come directly from gcc git.

This work was sponsored by `BA Robotic Systems'.

[Peter: use sizeof for character array sizes, make function static,
	extend commit message, add upstream gcc commit]
Signed-off-by: Jérôme Pouiller <jezz@sysmic.org>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2017-02-07 21:45:01 +01:00
Yann E. MORIN
3eccf76201 toolchain/wrapper: use the {str, len} tuples for the unsafe paths
In 61cb120 (toolchain/wrapper: extend paranoid check to -isystem), we
introduced a {str,len} tuple to check the various arguments pased to
gcc, to avoid hard-coding an ever-growing, long list of those args
directly in the condition check.

Now, we're left with a long list of unsafe paths, somehow hidden within
the code, which can use the same mechanism we use for arguments.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-12-11 15:45:06 +01:00
Yann E. MORIN
31c093e6d0 toolchain/wrapper: make the {str, len} tuple more generic, add comments
In 61cb120 (toolchain/wrapper: extend paranoid check to -isystem), we
introduced a {str,len} tuple to check the various arguments passed to
gcc, to avoid hard-coding an ever-growing, long list of those args
directly in the condition check.

It was made specific to the arguments (the structure member is named
'arg'), but can also be used to store the unsafe paths as well.

Also, that piece is almost un-documented.

Rename the structure member so that it is more generic, and add a bit of
comments to explain the whole of it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-12-11 15:44:44 +01:00
Vicente Olivert Riera
240564a693 arch/Config.in.mips: add support for XBurst cores
The Ingenic XBurst is a MIPS32R2 microprocessor.

It has a bug in the FPU that can generate incorrect results in certain
cases. The problem shows up when you have several fused madd
instructions in sequence with dependant operands.

Using the -mno-fused-madd option prevents gcc from emitting these
instructions. This patch adds changes to the toolchain wrapper to use
that option.

Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-11-09 21:36:34 +01:00
Yann E. MORIN
61cb120ee5 toolchain/wrapper: extend paranoid check to -isystem
Some packages, like libbsd, use -isystem flags to provide so-called
overrides to the system include files. In this particular case, this
is used in a .pc file, then used by antoher package; pkgconf does not
mangle this path; and eventually that other package ends up using
/usr/include/bsd to search for headers.

Our current toolchain wrapper is limited to looking for -I and -L, so
the paranoid check does not kick in.

Furthermore, as noticed by Arnout, there might be a bunch of other
so-unsafe options: -isysroot, -imultilib, -iquote, -idirafter, -iprefix,
-iwithprefix, -iwithprefixbefore; even -B and --sysroot are unsafe.

Extend the paranoid check to be able to check any arbitrary number of
potentially unsafe options:

  - add a list of options to check for, each with their length,
  - iterate over this list until we find a matching unsafe option.

Compared to previously, the list of options include -I and -L (which we
already had) extended with -idirafter, -iquote and -isystem, but leaving
all the others noticed by Arnout away, until we have a reason for
handling them.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-09-18 16:09:23 +02:00
Yann E. MORIN
105a8c156c toolchain/wrapper: display options leading to a paranoid failure
Current, we only display the path that causes the paranoid failure. This
is sufficient, as we can fail only for -I and -L options, and it is thus
easy to infer from the path, which option is the culprit.

However, we're soon to add a new test for the -isystem option, and then
when a failure occurs, we would not know whether it was because of -I or
-isystem. Being able to differentiate both can be hugely useful to
track down the root cause for the unsafe path.

Add two new arguments to the check_unsafe_path() function: one with the
current-or-previous argument, one to specify whether it has the path in
it or not. Print that in the error message, instead of just the path.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-09-18 16:07:35 +02:00
Ray Kinsella
968f5d5e59 arch/x86: add support for Intel X1000
The Intel X1000 is the Pentium class microprocessor that ships with
Galileo Gen 1/2. This patch adds changes to arch and toolchain-wrapper
to omit the lock prefix for the X1000.

[Thomas: tweak commit log and Config.in help text.]

Signed-off-by: Ray Kinsella <ray.kinsella@intel.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2015-10-20 10:04:52 +02:00
Arnout Vandecappelle
5ce73dca52 toolchain-external: bypass buildroot wrapper
The buildroot internal toolchain now adds a wrapper. When we use a
buildroot toolchain as an external toolchain, we want to bypass this
wrapper and call the compiler directly, for two reasons:

1. The options added by the wrapper are not necessarily appropriate
   when it is reused as an external toolchain. For instance, ccache
   may have been enabled while building the toolchain but not when
   using it as an external toolchain.

2. Currently, the wrapper expects to reside in .../usr/bin, but when
   used as an external toolchain it will be in .../ext-toolchain/bin.
   Therefore, the wrapper can't find the real binary and sysroot
   anymore.

To bypass the wrapper, we check for the existence of *.br_real files in
the external toolchain directory. If any such file exists, the wrapper
will add the .br_real suffix for all the wrapped files. Note that the
wrapper doesn't check if the *.br_real exists for each individual
wrapped file, it just assumes that all wrapped files have a
corresponding .br_real. This is currently true but that may change in
the future of course.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-17 10:50:35 +02:00
Peter Korsgaard
ccdb179d24 toolchain-wrapper.c: unbreak BR_CROSS_PATH_ABS handling
Fixes #8386

We should check if BR_CROSS_PATH_ABS is defined, not if it evalutates to
true for the pre processor.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-05 08:25:17 +02:00
Arnout Vandecappelle
1e97b27873 ccache: support changing the output directory
When building in a different output directory than the original build,
there will currently be a lot of ccache misses because in many cases
there is some -I/... absolute path in the compilation. Ccache has an
option CCACHE_BASEDIR to substitute absolute paths with relative paths,
so they wil be the same in the hash (and in the output).

Since there are some disadvantages to this path rewriting, it is made
optional as BR2_CCACHE_USE_BASEDIR. It defaults to y because the
usefulness of ccache is severely reduced without this option.

In addition to CCACHE_BASEDIR, we also substitute away the occurences
of $(HOST_DIR) in the calculation of the compiler hash. This is done
regardless of the setting of BR2_CCACHE_USE_BASEDIR because it's
quite harmless.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:21 +02:00
Arnout Vandecappelle
f4682cf933 ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain
Our current ccache disables hashing of the compiler executable itself,
because using the default 'mtime' doesn't work in buildroot: we always
rebuild the compiler, so the mtime is always different, so the cache
always misses.

However, in the current situation, if a user changes the compiler
configuration (which would result in the compiler generating different
object files than before) and does 'make clean all', ccache may in fact
reuse object files from the previous run. This rarely gives problems,
because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because
    different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical
    effect because the resulting code is usually still compatible,
(4) we currently don't use CCACHE_BASEDIR, and almost all object files
    will contain an absolute path (e.g. in debug info), so when
    building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only
    appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the
effects are really weird and hard to debug. Also, we want reproducible
builds and obviously the above makes builds non-reproducible. So we
have a FAQ entry that warns against using ccache and tells the user to
clear the cache in case of problems.

Now that ccache is called from the toolchain wrapper, it is in fact
possible to at least use the 'mtime' compiler hash for the external
toolchain and for the host-gcc. Indeed, in this case, the compiler
executable comes from a tarball so the mtime will be a good reference
for its state. Therefore, the patch (sed script) that changes the
default from 'mtime' to 'none' is removed.

For the internal toolchain, we can do better by providing a hash of
the relevant toolchain options. We are only interested in things that
affect the compiler itself, because ccache also processes the header
files and it doesn't look at libraries because it doesn't cache the
link step, just compilation. Everything that affects the compiler
itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of
course, also the compiler source itself is relevant, so the source
tarball and all the patches are included in the hash. For this purpose,
a new HOST_GCC_XTENSA_OVERLAY_TAR is introduced.

The following procedure tests the ccache behaviour:

 Use this defconfig:
BR2_arm=y
BR2_CCACHE=y

 make
 readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"

 Now make menuconfig, change variant into BR2_cortex_a9

 make clean; make
 readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
 should be "Cortex-A9"

After this commit, it is "Cortex-A9".

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Danomi Manchego <danomimanchego123@gmail.com>
Cc: Károly Kasza <kaszak@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:21 +02:00
Arnout Vandecappelle
792f1278e3 toolchain-wrapper: support change of BR2_CCACHE
By moving the ccache call to the toolchain wrapper, the following
scenario no longer works:

make foo-dirclean all BR2_CCACHE=

That's a sometimes useful call to check if some failure is perhaps
caused by ccache.

We can enable this scenario again by exporting BR_NO_CCACHE when
BR2_CCACHE is not set, and by handling this in the toolchain wrapper.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Romain Naour <romain.naour@openwide.fr>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:20 +02:00
Arnout Vandecappelle
d82f69cf10 infra: move ccache handling to the toolchain wrapper
Since we always have a toolchain wrapper now, we can move the ccache
call to the toolchain wrapper.

The hostcc ccache handling obviously stays.

The global addition of ccache to TARGET_CC/CXX is removed, but many
individual packages and infras still add it. This means we have a
chain like this: ccache -> toolchain-wrapper -> ccache -> gcc
However, this is fairly harmless: for cache misses, the inner ccache
just adds overhead and for cache hits, the inner ccache is never
called. Later patches will remove these redundant ccache calls.

As a side effect, perl now supports ccache as well.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Danomi Manchego <danomimanchego123@gmail.com>
Cc: Károly Kasza <kaszak@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:20 +02:00
Arnout Vandecappelle
919c06c282 gcc: use toolchain wrapper
We have a toolchain wrapper for external toolchain, but it is also
beneficial for internal toolchains, for the following reasons:

1. It can make sure that BR2_TARGET_OPTIMIZATION is passed to the
   compiler even if a package's build system doesn't honor CFLAGS.
2. It allows us to do the unsafe path check (i.e. -I/usr/include)
   without patching gcc.
3. It makes it simpler to implement building each package with a
   separate staging directory (per-package staging).
4. It makes it simpler to implement a compiler hash check for ccache.

The wrapper is reused from the external toolchain. A third CROSS_PATH_
option is added to the wrapper: in this case, the real executable is in
the same directory, with the extension .real.

The creation of the simple symlinks is merged with the creation of the
wrapper symlinks, otherwise part of the -gcc-ar handling logic would
have to be repeated.

The complex case-condition could be refactored with the one for the
external toolchain, but then it becomes even more complex because
they each have special corner cases. For example, the internal
toolchain has to handle *.real to avoid creating an extra indirection
after host-gcc-{final,initial}-rebuild.

Instead of creating the .real files, it would also have been possible
to install the internal toolchain in $(HOST_DIR)/opt, similar to what
we do for the external toolchain. However, then we would also have to
copy things to the sysroot and do more of the magic that the external
toolchain is doing. So keeping it in $(HOST_DIR)/usr/bin is much
simpler.

Note that gcc-initial has to be wrapped as well, because it is used for
building libc and we want to apply the same magic when building libc.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Jérôme Oufella <jerome.oufella@savoirfairelinux.com>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:20 +02:00
Arnout Vandecappelle
f6ae24379b toolchain-external: move wrapper to toolchain directory
The toolchain wrapper will be reused for the internal toolchain, so it
belongs in the toolchain directory. Also, the ext- prefix is removed
from it. The build commands are moved to a new toolchain-wrapper.mk.

The wrapper arguments that are also relevant for the internal toolchain
wrapper are moved to toolchain-wrapper.mk, the rest stays in
toolchain-external.mk.

While we're at it, move the building of the toolchain wrapper to the
build step of toolchain-external. There is no specific reason to do
this, other than that it fits better semantically. Also remove the
MESSAGE call, otherwise we'd see:
>>> toolchain-external undefined Building
>>> toolchain-external undefined Building toolchain wrapper
/usr/bin/gcc ...
Having an extra "Building toolchain wrapper' message is pointless.

The useless condition on $(BR2_TARGET_OPTIMIZATION) is removed. It was
always true because it wasn't qstrip'ped first, so clearly it works
without that condition as well.

Also rewrapped some comments and removed the 'external' reference.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Jérôme Oufella <jerome.oufella@savoirfairelinux.com>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2015-10-04 18:22:19 +02:00