kumquat-buildroot/docs/manual/common-usage.txt

// -*- mode:doc; -*-
// vim: set syntax=asciidoc:

== General Buildroot usage

include::make-tips.txt[]

include::rebuilding-packages.txt[]

=== Offline builds

If you intend to do an offline build and just want to download
all sources that you previously selected in the configurator
('menuconfig', 'nconfig', 'xconfig' or 'gconfig'), then issue:

--------------------
 $ make source
--------------------

You can now disconnect or copy the content of your +dl+
directory to the build-host.

=== Building out-of-tree

As default, everything built by Buildroot is stored in the directory
+output+ in the Buildroot tree.

Buildroot also supports building out of tree with a syntax similar to
the Linux kernel. To use it, add +O=<directory>+ to the make command
line:

--------------------
 $ make O=/tmp/build menuconfig
--------------------

Or:

--------------------
 $ cd /tmp/build; make O=$PWD -C path/to/buildroot menuconfig
--------------------

All the output files will be located under +/tmp/build+. If the +O+
path does not exist, Buildroot will create it.

*Note:* the +O+ path can be either an absolute or a relative path, but if it's
passed as a relative path, it is important to note that it is interpreted
relative to the main Buildroot source directory, *not* the current working
directory.

When using out-of-tree builds, the Buildroot +.config+ and temporary
files are also stored in the output directory. This means that you can
safely run multiple builds in parallel using the same source tree as
long as they use unique output directories.

For ease of use, Buildroot generates a Makefile wrapper in the output
directory - so after the first run, you no longer need to pass +O=<...>+
and +-C <...>+, simply run (in the output directory):

--------------------
 $ make <target>
--------------------

[[env-vars]]

=== Environment variables

Buildroot also honors some environment variables, when they are passed
to +make+ or set in the environment:

* +HOSTCXX+, the host C++ compiler to use
* +HOSTCC+, the host C compiler to use
* +UCLIBC_CONFIG_FILE=<path/to/.config>+, path to
  the uClibc configuration file, used to compile uClibc, if an
  internal toolchain is being built.
  +
  Note that the uClibc configuration file can also be set from the
  configuration interface, so through the Buildroot +.config+ file; this
  is the recommended way of setting it.
  +
* +BUSYBOX_CONFIG_FILE=<path/to/.config>+, path to
  the BusyBox configuration file.
  +
  Note that the BusyBox configuration file can also be set from the
  configuration interface, so through the Buildroot +.config+ file; this
  is the recommended way of setting it.
  +
* +BR2_CCACHE_DIR+ to override the directory where
  Buildroot stores the cached files when using ccache.
  +
* +BR2_DL_DIR+ to override the directory in which
  Buildroot stores/retrieves downloaded files.
  +
  Note that the Buildroot download directory can also be set from the
  configuration interface, so through the Buildroot +.config+ file. See
  xref:download-location[] for more details on how you can set the download
  directory.
* +BR2_GRAPH_ALT+, if set and non-empty, to use an alternate color-scheme in
  build-time graphs
* +BR2_GRAPH_OUT+ to set the filetype of generated graphs, either +pdf+ (the
  default), or +png+.
* +BR2_GRAPH_DEPS_OPTS+ to pass extra options to the dependency graph; see
  xref:graph-depends[] for the accepted options
* +BR2_GRAPH_DOT_OPTS+ is passed verbatim as options to the +dot+ utility to
  draw the dependency graph.
* +BR2_GRAPH_SIZE_OPTS+ to pass extra options to the size graph; see
  xref:graph-size[] for the acepted options

An example that uses config files located in the toplevel directory and
in your $HOME:

--------------------
 $ make UCLIBC_CONFIG_FILE=uClibc.config BUSYBOX_CONFIG_FILE=$HOME/bb.config
--------------------

If you want to use a compiler other than the default +gcc+
or +g+++ for building helper-binaries on your host, then do

--------------------
 $ make HOSTCXX=g++-4.3-HEAD HOSTCC=gcc-4.3-HEAD
--------------------

=== Dealing efficiently with filesystem images

Filesystem images can get pretty big, depending on the filesystem you choose,
the number of packages, whether you provisioned free space... Yet, some
locations in the filesystems images may just be _empty_ (e.g. a long run of
'zeroes'); such a file is called a _sparse_ file.

Most tools can handle sparse files efficiently, and will only store or write
those parts of a sparse file that are not empty.

For example:

* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks
  of sparse files:
** +tar cf archive.tar -S [files...]+ will efficiently store sparse files
   in a tarball
** +tar xf archive.tar -S+ will efficiently store sparse files extracted
   from a tarball

* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+,
  +never+ or +always+):
** +cp --sparse=always source.file dest.file+ will make +dest.file+ a
   sparse file if +source.file+ has long runs of zeroes

Other tools may have similar options. Please consult their respective man
pages.

You can use sparse files if you need to store the filesystem images (e.g.
to transfer from one machine to another), or if you need to send them (e.g.
to the Q&A team).

Note however that flashing a filesystem image to a device while using the
sparse mode of +dd+ may result in a broken filesystem (e.g. the block bitmap
of an ext2 filesystem may be corrupted; or, if you have sparse files in
your filesystem, those parts may not be all-zeroes when read back). You
should only use sparse files when handling files on the build machine, not
when transferring them to an actual device that will be used on the target.

=== Details about packages

[[package-details]]

Buildroot can produce a JSON blurb that describes the set of enabled
packages in the current configuration, together with their
dependencies, licenses and other metadata. This JSON blurb is produced
by using the +show-info+ make target:

------------------------
make show-info
------------------------

Buildroot can also produce details about packages as HTML and JSON
output using the +pkg-stats+ make target. Amongst other things, these
details include whether known CVEs (security vulnerabilities) affect
the packages in your current configuration. It also shows if there is
a newer upstream version for those packages.

------------------------
make pkg-stats
------------------------

=== Graphing the dependencies between packages

[[graph-depends]]

One of Buildroot's jobs is to know the dependencies between packages,
and make sure they are built in the right order. These dependencies
can sometimes be quite complicated, and for a given system, it is
often not easy to understand why such or such package was brought into
the build by Buildroot.

In order to help understanding the dependencies, and therefore better
understand what is the role of the different components in your
embedded Linux system, Buildroot is capable of generating dependency
graphs.

To generate a dependency graph of the full system you have compiled,
simply run:

------------------------
make graph-depends
------------------------

You will find the generated graph in
+output/graphs/graph-depends.pdf+.

If your system is quite large, the dependency graph may be too complex
and difficult to read. It is therefore possible to generate the
dependency graph just for a given package:

------------------------
make <pkg>-graph-depends
------------------------

You will find the generated graph in
+output/graph/<pkg>-graph-depends.pdf+.

Note that the dependency graphs are generated using the +dot+ tool
from the _Graphviz_ project, which you must have installed on your
system to use this feature. In most distributions, it is available as
the +graphviz+ package.

By default, the dependency graphs are generated in the PDF
format. However, by passing the +BR2_GRAPH_OUT+ environment variable, you
can switch to other output formats, such as PNG, PostScript or
SVG. All formats supported by the +-T+ option of the +dot+ tool are
supported.

--------------------------------
BR2_GRAPH_OUT=svg make graph-depends
--------------------------------

The +graph-depends+ behaviour can be controlled by setting options in the
+BR2_GRAPH_DEPS_OPTS+ environment variable. The accepted options are:

* +--depth N+, +-d N+, to limit the dependency depth to +N+ levels. The
  default, +0+, means no limit.

* +--stop-on PKG+, +-s PKG+, to stop the graph on the package +PKG+.
  +PKG+ can be an actual package name, a glob, the keyword 'virtual'
  (to stop on virtual packages), or the keyword 'host' (to stop on
  host packages). The package is still present on the graph, but its
  dependencies are not.

* +--exclude PKG+, +-x PKG+, like +--stop-on+, but also omits +PKG+ from
  the graph.

* +--transitive+, +--no-transitive+, to draw (or not) the transitive
  dependencies. The default is to not draw transitive dependencies.

* +--colors R,T,H+, the comma-separated list of colors to draw the
  root package (+R+), the target packages (+T+) and the host packages
  (+H+). Defaults to: +lightblue,grey,gainsboro+

--------------------------------
BR2_GRAPH_DEPS_OPTS='-d 3 --no-transitive --colors=red,green,blue' make graph-depends
--------------------------------

=== Graphing the build duration

[[graph-duration]]

When the build of a system takes a long time, it is sometimes useful
to be able to understand which packages are the longest to build, to
see if anything can be done to speed up the build. In order to help
such build time analysis, Buildroot collects the build time of each
step of each package, and allows to generate graphs from this data.

To generate the build time graph after a build, run:

----------------
make graph-build
----------------

This will generate a set of files in +output/graphs+ :

* +build.hist-build.pdf+, a histogram of the build time for each
  package, ordered in the build order.

* +build.hist-duration.pdf+, a histogram of the build time for each
  package, ordered by duration (longest first)

* +build.hist-name.pdf+, a histogram of the build time for each
  package, order by package name.

* +build.pie-packages.pdf+, a pie chart of the build time per package

* +build.pie-steps.pdf+, a pie chart of the global time spent in each
  step of the packages build process.

This +graph-build+ target requires the Python Matplotlib and Numpy
libraries to be installed (+python-matplotlib+ and +python-numpy+ on
most distributions), and also the +argparse+ module if you're using a
Python version older than 2.7 (+python-argparse+ on most
distributions).

By default, the output format for the graph is PDF, but a different
format can be selected using the +BR2_GRAPH_OUT+ environment variable. The
only other format supported is PNG:

----------------
BR2_GRAPH_OUT=png make graph-build
----------------

[[graph-size]]
=== Graphing the filesystem size contribution of packages

When your target system grows, it is sometimes useful to understand
how much each Buildroot package is contributing to the overall root
filesystem size. To help with such an analysis, Buildroot collects
data about files installed by each package and using this data,
generates a graph and CSV files detailing the size contribution of
the different packages.

To generate these data after a build, run:

----------------
make graph-size
----------------

This will generate:

* +output/graphs/graph-size.pdf+, a pie chart of the contribution of
  each package to the overall root filesystem size

* +output/graphs/package-size-stats.csv+, a CSV file giving the size
  contribution of each package to the overall root filesystem size

* +output/graphs/file-size-stats.csv+, a CSV file giving the size
  contribution of each installed file to the package it belongs, and
  to the overall filesystem size.

This +graph-size+ target requires the Python Matplotlib library to be
installed (+python-matplotlib+ on most distributions), and also the
+argparse+ module if you're using a Python version older than 2.7
(+python-argparse+ on most distributions).

Just like for the duration graph, a +BR2_GRAPH_OUT+ environment variable
is supported to adjust the output file format. See xref:graph-depends[]
for details about this environment variable.

Additionally, one may set the environment variable +BR2_GRAPH_SIZE_OPTS+
to further control the generated graph. Accepted options are:

* `--size-limit X`, `-l X`, will group all packages which individual
  contribution is below `X` percent, to a single entry labelled _Others_
  in the graph. By default, `X=0.01`, which means packages each
  contributing less than 1% are grouped under _Others_. Accepted values
  are in the range `[0.0..1.0]`.

* `--iec`, `--binary`, `--si`, `--decimal`, to use IEC (binary, powers
  of 1024) or SI (decimal, powers of 1000; the default) prefixes.

* `--biggest-first`, to sort packages in decreasing size order, rather
  than in increasing size order.

.Note
The collected filesystem size data is only meaningful after a complete
clean rebuild. Be sure to run +make clean all+ before using +make
graph-size+.

To compare the root filesystem size of two different Buildroot compilations,
for example after adjusting the configuration or when switching to another
Buildroot release, use the +size-stats-compare+ script. It takes two
+file-size-stats.csv+ files (produced by +make graph-size+) as input.
Refer to the help text of this script for more details:

----------------
utils/size-stats-compare -h
----------------

[[top-level-parallel-build]]
=== Top-level parallel build

.Note
This section deals with a very experimental feature, which is known to
break even in some non-unusual situations. Use at your own risk.

Buildroot has always been capable of using parallel build on a per
package basis: each package is built by Buildroot using +make -jN+ (or
the equivalent invocation for non-make-based build systems). The level
of parallelism is by default number of CPUs + 1, but it can be
adjusted using the +BR2_JLEVEL+ configuration option.

Until 2020.02, Buildroot was however building packages in a serial
fashion: each package was built one after the other, without
parallelization of the build between packages. As of 2020.02,
Buildroot has experimental support for *top-level parallel build*,
which allows some signicant build time savings by building packages
that have no dependency relationship in parallel. This feature is
however marked as experimental and is known not to work in some cases.

In order to use top-level parallel build, one must:

. Enable the option +BR2_PER_PACKAGE_DIRECTORIES+ in the Buildroot
configuration

. Use +make -jN+ when starting the Buildroot build

Internally, the +BR2_PER_PACKAGE_DIRECTORIES+ will enable a mechanism
called *per-package directories*, which will have the following
effects:

* Instead of a global _target_ directory and a global _host_ directory
  common to all packages, per-package _target_ and _host_ directories
  will be used, in +$(O)/per-package/<pkg>/target/+ and
  +$(O)/per-package/<pkg>/host/+ respectively. Those folders will be
  populated from the corresponding folders of the package dependencies
  at the beginning of +<pkg>+ build. The compiler and all other tools
  will therefore only be able to see and access files installed by
  dependencies explicitly listed by +<pkg>+.

* At the end of the build, the global _target_ and _host_ directories
  will be populated, located in +$(O)/target+ and +$(O)/host+
  respectively. This means that during the build, those folders will
  be empty and it's only at the very end of the build that they will
  be populated.

include::advanced.txt[]