mksquashfs allows to tweak many compressions parameters. Currently they
can't be changed from kmenu. Leaving out potential space savings.
This adds the option to enable a set of predetermined compression
options. This option is enabled by default for lz4 since lz4 currently
implicitly added the extreme to it in the makefile. So this aids in
keeping backward compatibility.
Signed-off-by: Linus Kaschulla <linus@cosmos-ink.net>
[yann.morin.1998@free.fr:
- fix check-package
- change the default code-style, add the legacy comment
- always add the qstriped string, as it's empty when not used
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
One advantage of squashfs over similar technologies is the support for
bigger block sizes. However the default size is not a lot bigger
(typically 128k if no `-b` flag specified).
This patch adds the ability to select from common block sizes
which for example can aid in improving compression ratio.
Signed-off-by: Linus Kaschulla <linus@cosmos-ink.net>
[yann.morin.1998@free.fr:
- drop spurious boolean-based setting in .mk
- split into multi-line
- qstrip variable expansion
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The prompt and variable name for the OCI "entrypoint arguments" are
somewhat incorrect. Indeed, they are in fact used to set the image
"command". Yet, using "command" would be confusing too, because the
interplay between entrypoint and command is tricky [0].
TL-DR; when both entrrypoint and command are set, command acts as
arguments passed to the entrypoint.
Additionally, we currently can only pass a single item as either
entrypoint or command. This precludes passing actual arguments to the
entrypoint, or passing multiple arguments as command.
For example:
BR2_TARGET_ROOTFS_OCI_ENTRYPOINT="/bin/tini -g -p SIGTERM --"
BR2_TARGET_ROOTFS_OCI_ENTRYPOINT_ARGS="/usr/bin/env sh"
generates an images with (only relevant fields are included below):
{
"config": {
"Entrypoint": [ "/bin/tini -g -p SIGTERM --" ],
"Cmd": [ "/usr/bin/env sh" ]
}
}
This is obviously incorrect, and not what one would expect:
{
"config": {
"Entrypoint": [ "/bin/tini", "-g", "-p", "SIGTERM", "--" ],
"Cmd": [ "/usr/bin/env", "sh" ]
}
}
However, some people do want to be able to pass an actual shell
scriptlet as a command, such as:
{
"config": {
"Entrypoint": [ "/bin/sh", "-c" ],
"Cmd": [ "my shell logic goes here" ]
}
}
Handling both is obviously conflicting: we can't both split-on-spaces
and not-split-on-spaces at the same time...
So, we fix that in two ways:
- make the current _OCI_ENTRYPOINT_ARGS a legacy option, and introduce
the new _OCI_CMD option with different semantics (see below) and an
appropriate prompt;
- we interpret both _OCI_ENTRYPOINT and _OCI_CMD as shell strings,
which we subject to the usual shell quoting [1] and token
recognition [2];
Since _OCI_ENTRYPOINT_ARGS used to be interpreted as a single string, we
can't easily change its meaning to be a space-separated list, as that
would break existing setups, which is the reason we make it legacy and
introduce a new option.
Ideally, we would like to default the new option _OCI_CMD to be the
quoted value of the previous _OCI_ENTRYPOINT_ARGS, but this is not
possible in Kconfig. Still, users that had a _OCI_ENTRYPOINT_ARGS set
will now get an early build error, and can still detect they need to do
something about it.
As for _OCI_ENTRYPOINT, it does not make much sense to support both cases.
Indeed, without splitting on spaces, we'd end up with an entrypoint that
would have a single item:
{
"config": {
"entrypoint: [ "some string with some spaces" ]
}
}
which in this case would try to execute the program which name is
actually "some string with some spaces", so we do not expect that
existing entrypoints are set with any space in them, and so the new
behaviour, unlike for _OCI_ENTRYPOINT_ARGS vs. _OCI_CMD, is compatible
with existing configurations, and so we do not need to make it a legacy
option and introduce a new one.
[0] https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact
[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02
[2] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_03
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Sergio Prado <sergio.prado@e-labworks.com>
Cc: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Add a dependency on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS to avoid
the following build failure when GO_GOARCH is empty (e.g. on mips32)
which leads to an empty --arch argument in the sloci-image call, raised
since the addition of the package in commmit
ccda2f4bdc:
printf ' rm -rf /home/autobuild/autobuild/instance-6/output-1/images/rootfs-oci\n /home/autobuild/autobuild/instance-6/output-1/host/bin/sloci-image --arch --entrypoint "sh" --author "Buildroot" --user "0" /home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/oci/target /home/autobuild/autobuild/instance-6/output-1/images/rootfs-oci:latest\n' >> /home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/oci/fakeroot
chmod a+x /home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/oci/fakeroot
PATH="/home/autobuild/autobuild/instance-6/output-1/host/bin:/home/autobuild/autobuild/instance-6/output-1/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl" FAKEROOTDONTTRYCHOWN=1 /home/autobuild/autobuild/instance-6/output-1/host/bin/fakeroot -- /home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/oci/fakeroot
rootdir=/home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/oci/target
table='/home/autobuild/autobuild/instance-6/output-1/build/buildroot-fs/full_devices_table.txt'
Usage:
sloci-image [options] ROOTFS NAME[:TAG]
sloci-image [-h | -V]
Create a single-layer OCI image with the given rootfs.
Arguments:
ROOTFS Directory or tar.gz archive with rootfs to pack into the image.
Important: Archive will be *moved* to the image, so make a copy if you
need it. Directory will be preserved.
NAME Name of the image.
TAG Tag for the image. Defaults to "latest".
Options:
-m --arch ARCH CPU architecture which the binaries in this image are built to run on.
Defaults to $(uname -m).
--arch-variant Variant of the CPU. This is typically used only for arm (v6, v7, v8).
-a --author NAME Name and/or email address of the person which created the image.
-c --cmd CMD Default arguments to the entrypoint of the container.
--debug Print debug messages (it can be also enabled with env. variable DEBUG).
-C --entrypoint EP Arguments to use as the command to execute when the container starts.
-e --env VAR=VAL Default environment variables for container.
-l --label KEY=VALUE Metadata for the container compliant with OCI annotation rules.
If KEY starts with a dot, it will be prefixed with
"org.opencontainers.image" (e.g. .url -> org.opencontainers.image.url).
--os OS Name of the OS which the image is built to run on. Defaults to "linux".
-p --port PORT[/PROT] Default set of ports to expose from a container running this image in
format: <port>/tcp, <port>/udp, or <port> (same as <port>/tcp).
Aliases: --expose.
-t --tar Pack image in a TAR archive.
-u --user USER The username or UID of user the process run as.
-v --volume PATH Default set of directories describing where the process is likely write
data specific to a container instance.
-w --working-dir DIR Sets the current working directory of the entrypoint process in the
container.
-V --version Print version and exit.
-h --help Print this message and exit.
Please report bugs at <https://github.com/jirutka/sloci-image/issues>.
make: *** [fs/oci/oci.mk:99: /home/autobuild/autobuild/instance-6/output-1/images/rootfs.oci] Error 1
Fixes:
- http://autobuild.buildroot.org/results/44da17a393421dfcb8bbdd63074cb82b436dfa94
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Different shells can have different behaviours when it comes to globbing
patterns. The dash shell (/bin/sh) on Debian testing switched to a
different fnmatch/glob implementation that results in this new behaviour:
Using bash:
$ mkdir /tmp/foo
$ echo /tmp/foo/.[^.]*
/tmp/foo/.[^.]*
Using dash:
$ mkdir /tmp/foo
$ echo /tmp/foo/.[^.]*
/tmp/foo/..
The current FAKEROOT script uses this shell glob pattern which now fails
on recent Debian testing systems:
rm: refusing to remove '.' or '..' directory: skipping '/build/buildroot-fs/cpio/target/run/..'
rm: refusing to remove '.' or '..' directory: skipping '/build/buildroot-fs/cpio/target/tmp/..'
Additionally, the glob will miss files which have at least two leading
dots, like ..foo ...bar or ......buz (highly improbable, but still).
It seems safer to use `find | xargs rm` here instead of relying on shell
globbing patterns.
Signed-off-by: Mathieu Mirmont <mat@parad0x.org>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
At run time, /run and /tmp get overmounted with a tmpfs, so anything
that is there becomes inaccessible.
Scripts in the fakeroot environment could call tools preparing the early
environment, leaving traces in /run or /tmp. For example, mkusers might
create home directories in /run: openssh sets the sshd home directory to
/run/sshd, so mkusers creates it. But since a tmpfs is mounted over it,
it doesn't exist at runtime, so the openssh service creates it at
startup (and deletes it when the service is stopped).
In addition, packages or rootfs overlay may leave things there as well.
Those may actually pose a runtime problem because the created file or
directory is missing - or it may not be a problem because the package
creates the missing files/directories on startup. In this situation,
it's better not to have them in the rootfs image (because they're not
functional anyway), but it's good to leave them in TARGET_DIR to make it
easier to debug the situation.
Therefore, remove the contents of /run and /tmp in the fakeroot
environment after ROOTFS_PRE_GEN_HOOKS, so they are still left in
TARGET_DIR.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
As reported by Jens [0], commit db7d786140 (fs/ext2: fix namespace for
variables) forgot to rename one variable expansion, resulting in the
ext2 label as set by the user to be ignored, with an empty label set.
[0] https://github.com/buildroot/buildroot/commit/db7d78614098#commitcomment-57918423
Reported-by: Jens Maus <mail@jens-maus.de>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add support for building an hybrid ISO9660 image compatible with legacy
and UEFI BIOS. Note that this is not about an (iso)hybrid image, which
can boot from both a CDROM or a USB stick, but really about an image
being bootable from the legay BIOS or EFI; the two are orthognal.
The option -eltorito-alt-boot need to be used in the xorriso command
to generate the hybrid image. That option is a separator, meaning the
previous boot entry is done, and the following boot options define a new
boot entry.
The -no-emul-boot defines the type of the current boot image; that's why
it has to now be repeated in each of the BIOS and EFI options.
Finally, for symetry and consistency between the BIOS and EFI options,
we move the BIOS image option first.
Note: the BIOS boot image options have to be provided before the EFI
ones, or the system won't boot; the underlying reason is not known...
Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
[yann.morin.1998@free.fr:
- note about hybrid vs. (iso)hybrid
- explain -eltorito-alt-boot
- explain duplication of -no-emul-boot
- rename the variables
- note about the BIOS-EFI ordering
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In commit 179ae068eb (fs/iso9660: add support to Grub EFI bootloader
in the image), we did a last-minute change when applying the patch, on
the flawed assumption that the commands were run in a standard Makefile
rule.
However, for filesystems, most commands are run in a script (so they run
under fakeroot). As such, we can't silence the commands with the usual
Makefile '@' construct.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add support to boot the Grub bootloader from an EFI BIOS in the ISO9660
image.
For that we need to create EFI System Partition (ESP). The ESP is a vfat
partition which contain the Grub2 binary at the /EFI/BOOT/ location.
xorriso command will generate the iso image including the ESP.
We notice Grub can not read and mount the ESP, therefore we place the Grub
configuration file in the ISO9660 partition. A Grub2 builtin configuration
need to be used to tell Grub2 to search automatically its configuration
file in the ISO9660 partition. Use 'set root=(cd0)' in the configuration
file passed to BR2_TARGET_GRUB2_BUILTIN_CONFIG_EFI.
Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
[yann.morin.1998@free.fr: fix timestamp fixup hook]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In filesystems, variables must be prefixed with ROOTFS_, to avoid
ckashing with packages of the same name.
We do not have a package named 'ext2', so we currently have no clash,
but it is still ebtter that the variables be properly namespaced.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Kory Maincent <kory.maincent@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In order to add support for EFI-compatible ISO9660 images in future
patches, this commit switch the ISO9660 logic to use xorriso instead of
cdrkit. Indeed the genimageiso tool from cdrkit doesn't have the
--efi-boot option needed to generate an image compatible with EFI BIOS.
Signed-off-by: Kory Maincent <kory.maincent@bootlin.com>
[yann.morin.1998@free.fr: drop superfluous tool name from variable]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add support to generate OCI (Open Container Initiative) images.
An OCI image consists of a manifest, an image index (optional), a set of
filesystem layers, and a configuration. The complete specification is
available in the link below:
https://github.com/opencontainers/image-spec/blob/master/spec.md
The image is generated with the host tool sloci-image, and config
options can be used to configure image parameters.
By default, the image is generated in a directory called rootfs-oci:
$ cd output/images
$ ls rootfs-oci/
blobs index.json oci-layout
Optionally, the image can be packed into a tar archive.
The image can be pushed to a registry using containers tools like
skopeo:
$ skopeo copy --dest-creds <user>:<pass> oci:rootfs-oci:<tag> \
docker://<user>/<image>[:tag]
And then we can pull/run the container image with tools like docker:
$ docker run -it <user>/<image>[:tag]
Signed-off-by: Sergio Prado <sergio.prado@e-labworks.com>
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
[Arnout:
- mention in help text that options are space separated;
- use GO_GOARCH and GO_GOARM for architecture;
- quote all arguments;
- don't cd to BINARIES_DIR;
- remove ROOTFS_OCI_IMAGE_NAME variable;
- remove wildcard from rm.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This enables EROFS big pcluster images for buildroot.
Signed-off-by: Gao Xiang <hsiangkao@aol.com>
[yann.morin.1998@free.fr: must be a multiple of 4KiB]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
U-Boot mkimage zstd is available since v2020.10:
26073f9ed3
Signed-off-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch sets the --with-xattr option when generating an image with
mkfs.jffs2, so that SELinux security contexts and other extended
attributes will be kept and available in jffs2 images.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The fakeroot script does not appear to be used in any of the checked
in defconfig targets, but it seems that most often the post
fakeroot script should be done after all the packages rules have
been applied instead of before.
Given that a change in systemd moved the SYSTEMD_PRESET_ALL hook to
a ROOTFS_PRE_CMD_HOOKS, there was no way to use a FAKEROOT script
to disable a service or fixup a systemd configuration. The systemd
move makes sense, and this just tries to preserve the same ability
to fixup a rootfs after all the cmd hooks are processed.
Refer to commit 65b63785a6 for
the change that instigated this reordering.
Signed-off-by: Charles Hardin <ckhardin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This patch is cosmetic and moves down ROOTFS_REPRODUCIBLE for
consistency.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Set the SELinux file security contexts using setfiles when generating
root filesystem images.
Without such security contexts created at build time, they need to be
setup at first boot by running the restorecon utility on the target.
This has two drawbacks:
- You have to special case the first boot, which cannot be done in
enforcing mode, and will have to run restorecon, then reboot.
- You cannot support read-only filesystems.
By setting up the security contexts at build time, we can have a
filesystem image that is immediately ready to boot an SELinux system
in enforcing mode, including if the root filesystem is read-only.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 98a6f1fc02 (fs/cpio: make initramfs init script survive 'console='
kernel argument) dropped the explicit /dev/console execs for fd 0,1,2, as
they fail when booted with console= and aren't really needed as the kernel
will setup fd 0,1,2 from /dev/console before executing the initramfs anyway.
Not doing this unfortunately confuses glibc's ttyname_r(3) implementation
(used by E.G. busybox/coreutils 'tty'), causing it to fail with ENOENT as
it does a fstat on fd 0 and tries to match up st_ino / st_dev against the
entries in /dev (since glibc 2.26):
commit 15e9a4f378c8607c2ae1aa465436af4321db0e23
Author: Christian Brauner <christian.brauner@canonical.com>
Date: Fri Jan 27 15:59:59 2017 +0100
linux ttyname and ttyname_r: do not return wrong results
If a link (say /proc/self/fd/0) pointing to a device, say /dev/pts/2, in a
parent mount namespace is passed to ttyname, and a /dev/pts/2 exists (in a
different devpts) in the current namespace, then it returns /dev/pts/2.
But /dev/pts/2 is NOT the current tty, it is a different file and device.
Detect this case and return ENODEV. Userspace can choose to take this as a hint
that the fd points to a tty device but to act on the fd rather than the link.
Signed-off-by: Serge Hallyn <serge@hallyn.com>
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
The reason it fails is that we manually mount devtmpfs on /dev in /init, so
the /dev/console used by the kernel (in rootfs) is not the same file as
/dev/console at runtime (in devtmpfs).
Notice: Once logged in, tty does work correctly. Presumably login reopens
stdin/stdout/stderr.
To fix this, re-add the exec of /dev/console for fd 0,1,2, but only do so if
possible. Because of the above mentioned shell behaviour (specified by
POSIX [0]), perform this check in a subshell.
[0] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_20_01
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The output of 'find' depends on the directory entries, and is not
ordered. As a consequence, the cpio archive is not reproducible.
Fix that by sorting the output of find. Use the 'C' locale to enforce
reproducibility that does not depend on the locale.
The command line is now pretty long, so we wrap it.
Signed-off-by: Yurii Monakov <monakov.y@gmail.com>
[yann.morin.1998@free.fr:
- use LC_ALL=C when sorting
- wrap long line
- reword commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Add an option to enable/disable padding to a multiple of 4k. Padding is
the default as it also was the past default behaviour.
Signed-off-by: Christoph Muellner <christoph.muellner@theobroma-systems.com>
Signed-off-by: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
It is often necessary to refer to other images that are present in
BINARIES_DIR from a custom ubinize configuration e.g. to include the
kernel in a UBI volume.
As we do for BR2_ROOTFS_UBIFS_PATH, replace BINARIES_DIR when copying
the file.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
[yann.morin.1998@free.fr:
- add the blurb in the help text
- rewrap commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When booting with 'console=<empty>' in the kernel command line (as e.g.
U-Boot does with silent flags in effect), opening /dev/console fails.
As per POSIX [0], when iany redirection fails, the shell running exec
shal exit in error. So, when 'console=<empty>' is specified.
/dev/console can't be opened, and the redirection fails, and /init is
killed.
That behaviour was fixed on the kernel side with commit 2bd3a997befc2
(Open /dev/console from rootfs), present since 2.6.34, released in May
2010, so any [dr]ecent kernel will have that fix.
Furthermore, busybox will fix things up anyway (in bb_sanitize_stdio()),
falling back to opening /dev/null if no console is availble. systemd
does a similar thing (in make_console_stdio()), and sysvinit again has
a similar approach (in console_init()).
The archealogy search turned up those relevant commits:
2011-08-04 10a130f91e initramfs/init: make sure that 0, 1, 2 fds are available
introduces the three exec redirections in initramfs
2011-09-06 3fac21ef8d cpio: fix boot with dynamic /dev
introduces the three exec redirections in cpio
2011-09-06 13a3afc536 fs/initramfs: refactor with fs/cpio
dropped the initramfs tweaks to reuse the cpio ones
2012-11-04 e1ebae700a fs/common: Create initial console device
introduces the /dev/console char,5,1 pseudo device creation in
cpio
2018-03-31 dec061adce fs/cpio: don't extend packages' permissions table
switched from the permission-table to a manual mknod to create
/dev/console
The redirections were added before we could guarantee there was a
/dev/console in the rootfs.
We're now guaranteed to have /dev/console in an initramfs, and any recent
kernel will automatically open /dev/console before spawning /init.
The three redirections are useless now, and cause harm under certain
conditions. Drop them.
[0] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_20_01
Signed-off-by: Timo Ketola <timo.ketola@exertus.fi>
Cc: Peter Korsgaard <peter@korsgaard.com>
[yann.morin.1998@free.fr:
- extend commit log with the analysis done with Peter
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently the volume-label for the root filesystem partition is a string
wit the following pattern: ad09a287-46a9-4790-ba97-fbbb549e5e96.
Specify the volume-label as "rootfs" to make it easier to identify it.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Fabio Estevam <festevam@gmail.com>
Tested-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This patch makes possible to create rootfs image using
EROFS filesystem.
Signed-off-by: Gao Xiang <hsiangkao@aol.com>
[yann.morin.1998@free.fr:
- simplify help text of filesystem entry
- drop the compression choice, keep the single boolean
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The initramfs is not a reall filesystem, so it does not use the
$(rootfs) infrastructure.
As a consequence, the usual rootfs-related variables are not set,
especially the name, type, and dependencies of the (non-)filesystem.
Yet, it is present in the list of rootfs to build, and thus we end
up including it in the output of show-info. But the missing variables
yield an incorrect json:
"": {
"type": "",
"virtual": false,
"version": "",
"licenses": "",
"dl_dir": "",
"install_target": ,
"install_staging": ,
"install_images": ,
"downloads": [ ],
"dependencies": [ ],
"reverse_dependencies": [ ]
},
First, the object key is empty; second, the install_target,
install_staging, and install_images values are empty, which is not
valid (if they were null, that be OK though). Third, this is clearly
the layout of a 'package' entry, not that of a 'rootfs' entry.
An option to fix that would be to actually make use of the rootfs
infra. However, that would mean doing a lot of work for nothing
(there is actually nothing to do, yet the infra would still do a lot
of preparatory and clean up work).
The alternative is pretty simple: declare and set the variables as if
it were a real filesystem, so that show-info can filter it to the
proper layout and can spit out appropriate content (even if fake).
The third option would be to teach show-info (and its internal
implementation, the macro json-info) to ignore specific cases, like
no-name items, or replace empty values with null, or whatnots. This
again would be quite a lot of work for a single occurence.
So we go for the simple faked variables.
We add linux as a dependency, so that the graph-depends also properly
represent the dependency chain, which ends up with something liKe:
ALL
|
v
rootfs-initramfs
| |
v v
linux rootfs-cpio
which is pretty fitting in the end.
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
Since this splits the file into blocks, the result will be not
bit-for-bit identical to single-threaded compression. Therefore, don't
enable this in BR2_REPRODUCIBLE builds.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Arnout: append the option instead of repeating the entire command]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Pass the recommended argument in the CPIO manual to make cpio archives
reproducible.
Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode
Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746
We can see that post-patch, the archive related differences are removed.
The differences are arising from utils/bin/getconf. This will have to
be investigated further. However, that is unrelated to cpio.
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Using the GZIP environment variable to pass gzip options is
deprecated, and therefore we are going to remove the "GZIP = -n"
definition from the main Buildroot Makefile. In preparation for this,
we explicitly add the -n argument to the gzip call in fs/common.mk to
ensure reproducibility.
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since we use --xattrs-include='*' to include all extended attributes,
tar creates a PAX formatted archive. The archive metadata captures atime
and ctime of files. To fix this, GNU recommends that we pass this added
argument to tar to create binary reproducible packages. Setting of mtime
is handled in fs/common.mk using touch on all files.
Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs
is created.
GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.
show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.
Beside, show-dependency-tree has never been part of a release.
Drop it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.
Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.
Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:
$ make {foo,bar}-show-info
{ "foo": { foo stuff } }
{ "bar": { bar stuff } }
However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):
$ make {foo,bar}-show-info |jq -s . -
[
{ "foo": { foo stuff } },
{ "bar": { bar stuff } }
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, only first-level dependencies of a filesystem are added to
the global list of packages, thus missing all recursive dependencies.
Use the newly introduced recursive variable instead, which already
contains the rootfs-common dependencies too.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This allows getting all the recursive dependencies of filesystems,
ike we have for packages, and allows us to treat both in a similar
fashion.
Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This makes the filesystems resemble packages yet a bit more, and will
allow sorting "items" on their type and names, when indexed from the
upper-case names.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
fakeroot by default forwards {f,l,}chown calls to libc and ignores
permission issues, which may cause issues when building in restricted
environments like user namespaces as set up with bubblewrap where a chown
call with a uid/gid not mapped in the user namespace instead returns EINVAL.
This error is not masked by fakeroot and returned to the caller, causing
failures.
There is no real reason to really perform the *chown calls in the context of
Buildroot (as the calls will likely just fail and files are not accessed
outside the fakeroot environment any way).
This forwarding can be disabled by setting the FAKEROOTDONTTRYCHOWN
environment variable, so set it when fakeroot is executed.
Reported-by: Esben Nielsen <nielsen.esben@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.
Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.
However, make being make, already builds the whole dependency tree
information, so we can just ask for it.
Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:
foo: target 1.2.3
foo -> bar host-meh
bar: target virtual
bar -> buz
buz: target 2.3.4
buz ->
host-meh: host virtual
host-meh -> host-bleark
host-bleark: host 3.4.5
host-bleark ->
rootfs-meh: host
rootfs-meh -> host-bleark
To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.
For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):
$ time make graph-depends
real 0m27.344s
$ time make show-dependency-tree
real 0m3.848s
>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):
$ time make graph-depends
real 1m56.459s
$ time make show-dependency-tree
real 0m5.748s
There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The static devices defined by packages are currently added to the full
device table when two conditions are met:
(1) ROOTFS_DEVICE_TABLES is non-empty
(2) BR2_ROOTFS_DEVICE_CREATION_STATIC=y
(2) is obviously correct. However, depending on (1) is not correct: if
the user doesn't provide any custom permission table and custom device
table, then ROOTFS_DEVICE_TABLES will be empty.
So instead, move the addition of the package-defined static devices
outside of condition (1), and have it only under condition (2).
Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In commit 6b50f988ad ("fs/common.mk:
rename internal variable"), USERS_TABLE was renamed to
ROOTFS_FULL_USERS_TABLE.
This commit follows the same direction by renaming the
FULL_DEVICE_TABLE variable to ROOTFS_FULL_DEVICE_TABLE.
In addition, for consistency, the file itself is renamed
full_device_table.txt.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In preparation of more renames, rename the variable that points to the
final users table.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: as suggested by Arnout, use ROOTFS_FULL_USERS_TABLE instead
of ROOTFS_FINAL_USERS_TABLE.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some filesystems may want to tweak their output names, rather than using
the fixed "rootfs.foo" scheme. Add a ROOTFS_FOO_IMAGE_NAME variable for
this purpose.
Signed-off-by: Carlos Santos <casantos@datacom.com.br>
[yann.morin.1998@free.fr: fix the patch]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.
For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.
This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.
Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.
Thanks to those changes, the global "dirs" target is entirely removed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
By default, tar will not include any extended attribute (xattr) when
creating archives, and thus will not store capabilties either (as they
are stored in the xattr 'security.capability').
Using option --xattrs is enough to create a tarball with all the xattrs
attached to a file. However, extracting all xattrs from a tarball
requires that --xattrs-include='*' be used. This is not symetric (but on
purpose, as per the documentation), and so is confusing to some.
So, we use --xattrs-include='*' to create the archive, so as to be
explicit that we want all xattrs to be stored.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The command "make show-build-order" doesn't show dependencies of rootfs-common target.
This patch adds $(ROOTFS_COMMON_DEPENDENCIES) to PACKAGES variable.
Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since 118534fe54 (fs: use a common tarball as base for the other
filesystems), the filesystem creation is split in two steps, using an
intermediate tarball to carry the generic, common finalisations to the
per-filesystem finalisation and image creation.
However, this intermediate tarball causes an issue with capabilities:
they are entirely missing in the generated filesystems.
Capabilities are stored in the extended attribute security.capability,
which tar by default will not store/restore, unless explicitly told to,
e.g. with --xattrs-include='*', which we don't pass.
Now, passing this option when creating and extracting the intermediate
tarball, both done under fakeroot, will cause fakeroot to report an
invalid filetype for files with capabilities. mksquashfs would report
such unknown files as a warning, while mkfs.ext2 would fail (with a
similar error message), e.g.:
File [...]/usr/sbin/getcap has unrecognised filetype 0, ignoring
This is due to a poor interaction between tar and fakeroot; running as
root the exact same commands we run under fakeroot, works as expected.
Unfortunately, short of fixing fakeroot (which would first require
understanding the problem in there), we don't have much options.
The intermediate tarball was made to avoid redoing the same actions over
and over again for each filesystem to build. However, most of the time,
only one or two such filesystems would be enabled [0], and those actions
are usually pretty lightweight. So, using an intermediate tarball does
not provide a big optimisation.
The main reason to introduce the intermediate tarball, however, is that
it allows to postpone per-filesystem finalisations to be applied only
for the corresponding filesystem, not for all of them.
So, we get rid of the intermediate tarball, and simply move all of the
code to run under fakeroot to the per-filesystem fakeroot script.
Instead of extracting the intermediate tarball, we just rsync the
original target/ directory, and apply the filesystem finalisations on
that copy. The only thing still done in the rootfs-common step is to
generate the intermediate files (users file, devices file) that are used
in the fakeroot script.
Fixes: https://bugs.busybox.net/show_bug.cgi?id=11216
Note: an alternate solution would have been to keep the intermediate
tarball to keep most of the common finalisations, and move only the
permissions to each filesystem, but that was getting a bit more complex
and changed the ordering of permissions and post-fakeroot scripts. Once
we bite the bullet of having some common finalisation done in each
filesystem, it's easier to just move all of them.
[0] Most probsably, users would enable the real filesystem to put on
their device, plus the 'tar' filesystem, to be able to easily inspect
the content on their development machine.
Reported-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
