Commit Graph

1016 Commits

Author SHA1 Message Date
Andreas Naumann
f08a7f3fc5 package/pkg-qmake: new qmake package infrastructure
This provides generic functions for Qt5 qmake based packages. It will
make it possible to remove lots of redefinition of
QT5_xxx_{CONFIGURE|BUILD|INSTALL_STAGING}_CMDS. Additionally it
provides a generic target install method which will make most of the
package specific commands obsolete.

This is done by re-running the install step of the qmake generated
Makefile with the package build directory prepended (to the
staging/host path). Even though this does create lengthy pathes it
allows for easy separation of the staging files from the host destined
files by just omitting the resulting BUILD_DIR+HOST_DIR path from the
following rsync call to the real target folder.  The cleanup of many
files we dont want in target is deferred to the target-finalize
step. In addition to what's being removed already, we also have to
cleanup some Qt5 specific files (prl) and the documentation directory.

This approach was chosen over copying all files recorded in the pkg-files-list
after some discussion which Thomas Petazzoni summed up:
"We don't yet use pkg-files-list really as part of the build
process anywhere, I feel a bit more comfortable at this point with what
Andreas is proposing."

Thanks to this infrastructure, it will be possible to get rid of the
many conditional install commands because qmake already takes care of
this when generating the Makefile install targets with the given or
autodetected configure options of each package.

However, custom install steps may have to remain in cases where a
particular Buildroot option has no corresponding setting in the
packages configuration options.

Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-09 22:26:42 +01:00
Peter Korsgaard
b806ded575 Kickoff 2020.05 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-09 10:36:20 +01:00
Peter Korsgaard
5f5477b0ab Update for 2020.02
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-08 22:23:03 +01:00
Yann E. MORIN
9e2128bf50 Makefile: work around a bug in newly released make 4.3
Several users of rolling-release distributions have been reporting on
IRC that Buildroot is broken now that they have switched to the newly
released make 4.3.

It turns out that the constructs we use to generated and include the
internal br2-external related fragments is no longer working with
make-4.3.

Indeed, an upstream bug report [0] seems to imply that it so far was
working by chance. There has been no further feedback, whether this is
really considered a fix for a previous ill-defined behaviour, or an
actual regression...

In the meantime, we add a workaround, suggested in that same bug report,
that fixes the issue for make 4.3, and that should not break on older
make versions either (verified on all relevant versions: from 3.81,
3.82, 4.0, 4.1, and 4.2).

[0] https://savannah.gnu.org/bugs/?57676

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Mircea Gliga <mgliga@bitdefender.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-05 16:39:53 +01:00
Thomas Petazzoni
98e11e8c8f Makefile: remove bogus comment
The comment "Check files that are touched by more than one package"
was previously located right before the calls to the check-uniq-files
script. However, this script and the logic calling it have been
removed in commit 2496189a42 ("core:
drop check-uniq-files"), so the comment no longer makes any sense:
let's drop it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-03 22:30:14 +01:00
Peter Korsgaard
338e2b5ba1 Update for 2020.02-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-03 00:03:08 +01:00
Thomas Petazzoni
0e2be4db8a package/pkg-generic: make file list logic parallel build compatible
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.

This works well when packages are built sequentially, but badly fails
when using top-level parallel build.

More specifically, top-level parallel build can fail with:

comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory

Because that file has been removed concurrently by the build process
of another package.

This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.

At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.

There are however some differences:

 (1) The files are no longer ordered in build order, but by alphabetic
     ordering of packages. Indeed, "build order" no longer makes any
     sense in the context of top-level parallel build.

 (2) Some files which were incorrectly tracked are no longer
     tracked. For example, the toolchain package is a target package,
     but it installs files in $(HOST_DIR). In the previous logic, the
     files installed by the toolchain package in $(HOST_DIR) were
     incorrectly affected to the next host package that was installed
     after the toolchain package. With our new logic, those files are
     no longer tracked at all. To fix this, we would have to change
     the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
     installation steps, not just for the install-host, install-target
     and install-staging steps respecitively. But the result was
     already incorrect anyway, and therefore this should be fixed
     separately.

Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).

Fixes:

  http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-27 22:46:20 +01:00
Peter Korsgaard
b36e2eca20 Update for 2020.02-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-26 17:18:34 +01:00
Romain Naour
1776190fb1 Config.in: add BR2_HOST_GCC_AT_LEAST_9
Fedora 30 switched to GCC 9.x. [1]

[1] https://fedoraproject.org/wiki/Changes/GCC9

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-25 23:21:50 +01:00
Yann E. MORIN
ea8dd081aa infra: don't be verbose when calling the instrumentation steps
Commit 509db3b88a added calls to (parts of) the instrumentation steps.
However, those calls are echoed, unlike the other places where we call
them (in the package infra).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Acked-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
2020-02-22 22:29:34 +01:00
Thomas De Schampheleire
9b82442314 Makefile: don't recreate staging symlink if it exists
Create the staging symlink the same way as the host symlink. This means
using a make dependency rather than recreating it every time.

In coreutils versions below 8.27, re-creation of symbolic links was not
atomic. This means that there is a period in time where the existing link is
removed, before the new one is created. In coreutils 8.27 this was fixed,
see [1]. Note that CentOS 7 ships with coreutils 8.22.

In the following scenario, this is a problem:

- an application is compiled using the sysroot prepared by Buildroot and
  links against Xenomai userspace libraries, but its build process is steered
  from outside of Buildroot
- to know the correct flags, the application makefile uses the 'xeno-config'
  file to request them, and passes DESTDIR=/buildroot/output/staging
- the xeno-config responds with flags based on the path
  '/buildroot/output/staging/...'
- while the application build is ongoing, a 'make' happens in Buildroot,
  causing the 'staging' symlink to be recreated (even though it already
  existed)
- when exactly at this time, the application calls the compiler with -I
  flags pointing to output/staging, the build fails with:

  -I/buildroot/output/staging/usr/include/xenomai/mercury: Error:  ^ is not a directory
  -I/buildroot/output/staging/usr/include/xenomai: Error:  ^ is not a directory
  -I/buildroot/output/staging/usr/include/xenomai/xenomai: Error:  ^ is not a directory
  -I/buildroot/output/staging/usr/include/xenomai/psos: Error:  ^ is not a directory
  Failed: ** ^ *

Work around this problem by only creating the staging symlink once, similar
to how the host symlink (if any) is created.

See also commit d0f4f95e39 which changed the
way these symlinks are made. The reasoning in this commit is to move away
from the 'dirs' target.

[1] 376967889e

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-19 20:13:57 +01:00
Thomas De Schampheleire
1b62227d43 Makefile: use HOST_DIR_SYMLINK instead of hardcoding
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-19 20:13:50 +01:00
Peter Korsgaard
22f07ab2b5 Update for 2020.02-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-18 23:31:02 +01:00
Thomas De Schampheleire
509db3b88a core: fix packages-file-list.txt after an incremental build
The package instrumentation step 'step_pkg_size' is populating the files:
    output/build/packages-file-list.txt
    output/build/packages-file-list-staging.txt
    output/build/packages-file-list-host.txt
by comparing the list of files before and after installation of a package,
with some clever tricks to detect changes to existing files etc.

As an optimization, instead of gathering this list before and after each
package, where the 'after-state' of one package is the same as the
'before-state' of the next package, only the 'after-state' is used and
is shared between packages.

This works fine, except at the end of the build, as explained next.

In the target-finalize step, many files will be touched. For example, files
like /etc/hosts, /etc/os-release, but also all object files that are
stripped, and all files touched by post-build scripts or created by rootfs
overlays. This means that the 'after-state' of the last package does not
reflect the actual situation after target-finalize is run.

For a single complete build this poses no problem. But, if one incrementally
rebuilds a package after the initial build, e.g. with 'make foo-rebuild',
then all changes that happened in target-finalize at the end of the initial
build (the 'after-state' of the last package built) will be detected as
changes caused by the rebuild of package foo. As a result, all these files
will incorrectly be treated as 'owned' by package foo.

Correct this situation by capturing a new state at the end of
target-finalize, so that the 'before-state' of an incremental build will be
correct.

Note: the reasoning above talks about packages-file-list.txt and
target-finalize, but also applies to
packages-file-list-staging.txt/staging-finalize and
packages-file-list-host.txt/host-finalize.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-17 22:19:33 +01:00
Peter Korsgaard
6648cfc749 Makefile, manual, website: Bump copyright year
Happy 2020!

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-01-29 22:31:02 +01:00
Peter Korsgaard
7b24bd59c8 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-02 09:39:41 +01:00
Peter Korsgaard
14ec220df3 Kickoff 2020.02 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-02 08:57:02 +01:00
Peter Korsgaard
836b84a774 Update for 2019.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-01 22:39:47 +01:00
Thomas Petazzoni
d9a4c0c7ea Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y
With per-package folder support, top-level parallel build becomes
safe, so we can enclose the .NOTPARALLEL statement in a
!BR2_PER_PACKAGE_DIRECTORIES condition.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 15:13:45 +01:00
Thomas Petazzoni
c4e6d5c8be core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.

There are two main benefits:

 - Packages will now see only the dependencies they explicitly list in
   their <pkg>_DEPENDENCIES variable, and the recursive dependencies
   thereof.

 - We can support top-level parallel build properly, because a package
   only "sees" its own host directory and target directory, isolated
   from the build of other packages that can happen in parallel.

It works as follows:

 - A new output/per-package/ directory is created, which will contain
   one sub-directory per package, and inside it, a "host" directory
   and a "target" directory:

   output/per-package/busybox/target
   output/per-package/busybox/host
   output/per-package/host-fakeroot/target
   output/per-package/host-fakeroot/host

   This output/per-package/ directory is PER_PACKAGE_DIR.

 - The global TARGET_DIR and HOST_DIR variable now automatically point
   to the per-package directory when PKG is defined. So whenever a
   package references $(HOST_DIR) or $(TARGET_DIR) in its build
   process, it effectively references the per-package host/target
   directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
   is handled as well.

 - Of course, packages have dependencies, so those dependencies must
   be installed in the per-package host and target directories. To do
   so, we simply rsync (using hard links to save space and time) the
   host and target directories of the direct dependencies of the
   package to the current package host and target directories.

   We only need to take care of direct dependencies (and not
   recursively all dependencies), because we accumulate into those
   per-package host and target directories the files installed by the
   dependencies. Note that this only works because we make the
   assumption that one package does *not* overwrite files installed by
   another package.

   This is done for "extract dependencies" at the beginning of the
   extract step, and for "normal dependencies" at the beginning of the
   configure step.

This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:

 - The filesystem image creation code cannot work.

 - We don't have a SDK to build code outside of Buildroot.

In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.

This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:

 - The check-host-rpath script, which verifies at the end of each
   package installation that it has the appropriate RPATH, is modified
   to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
   a correct RPAT.

 - The fix-rpath script, which mungles the RPATH mainly for the SDK
   preparation, is modified to rewrite the RPATH to not point to
   per-package directories. Indeed the patchelf --make-rpath-relative
   call only works if the RPATH points to the ROOTDIR passed as
   argument, and this ROOTDIR is the global host directory. Rewriting
   the RPATH to not point to per-package host directories prior to
   this is an easy solution to this issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 14:24:05 +01:00
Peter Korsgaard
fc1c7e5961 Update for 2019.11-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-24 11:15:16 +01:00
Peter Korsgaard
1ab7e0c6f3 Update for 2019.11-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-16 23:02:26 +01:00
Peter Korsgaard
78d373f787 Update for 2019.11-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-05 23:39:17 +01:00
Yann E. MORIN
2496189a42 core: drop check-uniq-files
Back a few years ago, when we were starting to think about top-level
parallel build, we were not sure how to deal with packages that
installed the same files, so we wanted to catch the situation to assess
how prevalent that was, before we decided what to do and how to address
it.

However, the trend nowadays is that packages will install in a
per-package target/ (and staging/ and host/), and the final directories
will be assembled in a reproducible (alphabetical) order, so if two
packages install the same file, the last one will win (as is currently
the case).

Besides, check-uniq-files reports loads of spurious errors when packages
get reinstalled (e.g. during development).

Finally, check-uniq-files is the only script called during the build,
that is written in python.

So, get rid of check-uniq-files.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-26 21:19:07 +02:00
Peter Korsgaard
d8e6851f11 Merge branch 'next' 2019-09-03 15:03:02 +02:00
Peter Korsgaard
ba4d142886 Kickoff 2019.11 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-09-02 22:54:38 +02:00
Peter Korsgaard
1fcdfbfb8a Update for 2019.08
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-09-01 23:06:01 +02:00
Peter Korsgaard
cd8ab1853d Update for 2019.08-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-28 23:02:48 +02:00
Yann E. MORIN
e9cdabee71 support/graph-size: add option to change percentage to group in Others
Currently, we group packages that contribute less then 1%, into the
"Other" category.

However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.

Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.

Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
 - remove empty base class definition from Config;
 - use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:44:27 +02:00
Peter Korsgaard
f3221f1abf Update for 2019.08-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-20 14:39:52 +02:00
Peter Korsgaard
9b9abb0dd0 Update for 2019.08-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-09 09:04:28 +02:00
Brent Generous
d57e73078a Makefile: ensure $BINARIES_DIR exist before post-image scripts
When no filesystem is enabled, the $BINARIES_DIR is not created. Yet,
the post-image scripts are still run. When those want to generate an
image in there, they may fail as the dirctory does not exist (it did
exist before we started applying preparatory changes for top-level
parallel build, so scripts got to rely on that assumption).

Do in target-post-image as we do in the sdk rule: create the directory
before calling the scripts.

Signed-off-by: Brent Generous <bgenerous@impinj.com>
[yann.morin.1998@free.fr:
  - create the directory before calling the scripts
  - don't drop the creation in the sdk rule
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 17:37:56 +02:00
Yann E. MORIN
f879203cfc core: drop now-useless prepare-kconfig rule
This rule was added back in 9429e7b698 (core: introduce an intermediate
rule before the configurators) when the kconfig-side br2-external file
was generated separately from the Makefile-side one.

Now that they are generated together very early in the Makefile, we no
longer need this intermediate rule. Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
[Peter: also drop outdated reference in the manual]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
d027cd75d0 core: generate all br2-external files in one go
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.

But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.

And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.

Make it so.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
76345bf6bc core: simplify removal of generated br2-external files
Now that all the br2-external generated files are named after the same
pattern, it gets easier to remove them all using a glob.

Furthermore, we're on the verge of introducing more such generated
files, so removing them at one fell swoop will be simpler too.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
492d09bab2 core: rename generated .br-external.mk file
Now that the two (all of them!) br2-external related files are generated
in the same location, it makes sense they are named after the same
pattern.

When initial support for (then single) br2-external trees was added back
in a4239f7fd1 (core: introduce the BR2_EXTERNAL variable), it was not
clear-cut why that file was not named with a br2 prefix.

So rename it now.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN
54af0551b8 core: move generated .br2-external kconfig file to $(BASE_DIR)
Currently, that file is generated rather late in the configuration
process, so BUILD_DIR is known (and exists) by then.

We're soon to generate that file much earlier, at a point where
BUILD_DIR is not yet known, so we have two options:
 1- declare BUILD_DIR earlier;
 2- generate the file in an already-known location.

We go with the second solution, as we're already generating a
br2-external related file in BASE_DIR, so we can as well generate all
br2-external files in the same place.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Atharva Lele
d590b37633 Makefile: don't export GZIP environment variable
We export GZIP = -n so that GZIP does not record original
name and timestamps. However..

GZIP environment variable is deprecated and soon will not be
supported in future GZIP versions. GZIP suggests the use of a
wrapper to pass options globally but it might be difficult to
implement in Buildroot. For now, we don't export the variable
and fix reproducibility issues per package as they show up in
Autobuilder.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-07-17 08:46:05 +02:00
Peter Korsgaard
2421c246b2 Kickoff 2019.08 cycle
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 10:17:27 +02:00
Peter Korsgaard
8d4e26da08 Update for 2019.05
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 10:01:07 +02:00
Peter Korsgaard
2c96d648a2 Update for 2019.05-rc3
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-25 23:38:05 +02:00
Peter Korsgaard
2adc578dfd Update for 2019.05-rc2
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-15 22:35:04 +02:00
Peter Korsgaard
a8b229a75c Update for 2019.05-rc1
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-08 22:54:44 +02:00
Yann E. MORIN
f5f17c4dd7 core: remove show-dependency-tree
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.

show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-07 23:06:05 +02:00
Nylon Chen
b58a6bf774 package/binutils: fix build error due to architecture name is incomplete
Fixes
  http://autobuild.buildroot.net/results/128/12803a705586e82fdfb49013da2eb3b9879ccd45/

Signed-off-by: Che-Wei Chuang <cnoize@andestech.com>
Signed-off-by: Greentime Hu <greentime@andestech.com>
Signed-off-by: Nylon Chen <nylon7@andestech.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-20 16:16:39 +02:00
Yann E. MORIN
5abb88218e core: add per-package and per-filesystem show-info
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:47:22 +02:00
Yann E. MORIN
f8177b7813 core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.

Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...

We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).

The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.

It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma:  [ { ... }, { ... } ]

So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).

Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:37:44 +02:00
Peter Korsgaard
c24faa81e8 Makefile: release: really drop build/docs from release tarball
Commit 15cb98769e (release: remove manual build files from release
tarballs) tried to remove the temporary files from the manual build from the
release tarball, but manual-clean only removes build/docs/manual and leaves
build/docs in the tarball.

Instead use 'make clean' to completely remove the build directory from the
tarball.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-29 23:09:41 +01:00
Yann E. MORIN
870f37fe04 core: add make-based full-dependency list
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.

Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.

However, make being make, already builds the whole dependency tree
information, so we can just ask for it.

Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:

    foo: target 1.2.3
    foo -> bar host-meh
    bar: target virtual
    bar -> buz
    buz: target 2.3.4
    buz ->
    host-meh: host virtual
    host-meh -> host-bleark
    host-bleark: host 3.4.5
    host-bleark ->
    rootfs-meh: host
    rootfs-meh -> host-bleark

To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.

For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):

    $ time make graph-depends
    real    0m27.344s
    $ time make show-dependency-tree
    real    0m3.848s

>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):

    $ time make graph-depends
    real    1m56.459s
    $ time make show-dependency-tree
    real    0m5.748s

There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 15:39:12 +01:00
Arnout Vandecappelle (Essensium/Mind)
fd5bd12379 Makefile: printvars: don't print anything when VARS is not set
Using 'make printvars' for printing all variables is not very useful.
E.g. all macros will output some bogus value. In addition, the same can
be achieved with 'make -p'.

We can simply remove the condition on $(VARS). If VARS is not set, the
filter expression will be empty which matches nothing, so nothing is
printed.

Note that the old behaviour can still be achieved with:
make printvars VARS=%

Update the 'make help' text to match the new behaviour.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-12 21:58:32 +01:00