Commit Graph

327 Commits

Author SHA1 Message Date
Yann E. MORIN
9bdc27d8a7 fs: don't use := when not needed
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-27 10:35:06 +01:00
James Hilliard
36fb6d174d fs/common.mk: enable multithreaded xz compression
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM   use at most NUM threads; the default is 1; set to 0
                    to use as many threads as there are processor cores

Since this splits the file into blocks, the result will be not
bit-for-bit identical to single-threaded compression. Therefore, don't
enable this in BR2_REPRODUCIBLE builds.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Arnout: append the option instead of repeating the entire command]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 19:29:47 +02:00
Thomas Petazzoni
c4d09c8b18 fs/cpio: fix typo in comment
--reproducible is --reproducible, not --repoducible.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-07-17 08:56:35 +02:00
Atharva Lele
fd544aaebb fs/cpio: make cpio rootfs reproducible
Pass the recommended argument in the CPIO manual to make cpio archives
reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed.
The differences are arising from utils/bin/getconf. This will have to
be investigated further. However, that is unrelated to cpio.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-07-17 08:53:03 +02:00
Atharva Lele
44d17dd128 fs/common.mk: do not store original names and timestamps when creating gzipped rootfs
Using the GZIP environment variable to pass gzip options is
deprecated, and therefore we are going to remove the "GZIP = -n"
definition from the main Buildroot Makefile. In preparation for this,
we explicitly add the -n argument to the gzip call in fs/common.mk to
ensure reproducibility.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-07-17 08:45:11 +02:00
Atharva Lele
9c449f9dd8 fs/tar: explicitly set extended header values to ensure binary reproducibility
Since we use --xattrs-include='*' to include all extended attributes,
tar creates a PAX formatted archive. The archive metadata captures atime
and ctime of files. To fix this, GNU recommends that we pass this added
argument to tar to create binary reproducible packages. Setting of mtime
is handled in fs/common.mk using touch on all files.

Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs
is created.

GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-07-14 12:21:11 +02:00
Yann E. MORIN
f5f17c4dd7 core: remove show-dependency-tree
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.

show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-07 23:06:05 +02:00
Yann E. MORIN
5abb88218e core: add per-package and per-filesystem show-info
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:47:22 +02:00
Yann E. MORIN
2b35c71745 fs: add all recursive dependencies to packages list
Currently, only first-level dependencies of a filesystem are added to
the global list of packages, thus missing all recursive dependencies.

Use the newly introduced recursive variable instead, which already
contains the rootfs-common dependencies too.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:36:25 +02:00
Yann E. MORIN
d62e46ac77 fs: introduce variable with all recursive dependencies
This allows getting all the recursive dependencies of filesystems,
ike we have for packages, and allows us to treat both in a similar
fashion.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:29:57 +02:00
Yann E. MORIN
4d83c72370 fs: introduce variables with name and type
This makes the filesystems resemble packages yet a bit more, and will
allow sorting "items" on their type and names, when indexed from the
upper-case names.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:28:55 +02:00
Peter Korsgaard
655acd1df0 fs/common.mk: disable real chown calls in fakeroot
fakeroot by default forwards {f,l,}chown calls to libc and ignores
permission issues, which may cause issues when building in restricted
environments like user namespaces as set up with bubblewrap where a chown
call with a uid/gid not mapped in the user namespace instead returns EINVAL.
This error is not masked by fakeroot and returned to the caller, causing
failures.

There is no real reason to really perform the *chown calls in the context of
Buildroot (as the calls will likely just fail and files are not accessed
outside the fakeroot environment any way).

This forwarding can be disabled by setting the FAKEROOTDONTTRYCHOWN
environment variable, so set it when fakeroot is executed.

Reported-by: Esben Nielsen <nielsen.esben@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-04-06 19:12:09 +02:00
Yann E. MORIN
870f37fe04 core: add make-based full-dependency list
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.

Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.

However, make being make, already builds the whole dependency tree
information, so we can just ask for it.

Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:

    foo: target 1.2.3
    foo -> bar host-meh
    bar: target virtual
    bar -> buz
    buz: target 2.3.4
    buz ->
    host-meh: host virtual
    host-meh -> host-bleark
    host-bleark: host 3.4.5
    host-bleark ->
    rootfs-meh: host
    rootfs-meh -> host-bleark

To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.

For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):

    $ time make graph-depends
    real    0m27.344s
    $ time make show-dependency-tree
    real    0m3.848s

>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):

    $ time make graph-depends
    real    1m56.459s
    $ time make show-dependency-tree
    real    0m5.748s

There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 15:39:12 +01:00
Thomas Petazzoni
ac3b476bf5 fs/common.mk: make sure that static devices from packages are created
The static devices defined by packages are currently added to the full
device table when two conditions are met:

 (1) ROOTFS_DEVICE_TABLES is non-empty
 (2) BR2_ROOTFS_DEVICE_CREATION_STATIC=y

(2) is obviously correct. However, depending on (1) is not correct: if
the user doesn't provide any custom permission table and custom device
table, then ROOTFS_DEVICE_TABLES will be empty.

So instead, move the addition of the package-defined static devices
outside of condition (1), and have it only under condition (2).

Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-12-04 21:53:14 +01:00
Yann E. MORIN
b67f0a99a9 fs/common.mk: allow user provided permissions to override packages permissions
Reported-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-12-04 21:27:01 +01:00
Thomas Petazzoni
93434aedaa fs/common.mk: rename FULL_DEVICE_TABLE to ROOTFS_FULL_DEVICES_TABLE
In commit 6b50f988ad ("fs/common.mk:
rename internal variable"), USERS_TABLE was renamed to
ROOTFS_FULL_USERS_TABLE.

This commit follows the same direction by renaming the
FULL_DEVICE_TABLE variable to ROOTFS_FULL_DEVICE_TABLE.

In addition, for consistency, the file itself is renamed
full_device_table.txt.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-12-04 21:26:39 +01:00
Yann E. MORIN
6b50f988ad fs/common.mk: rename internal variable
In preparation of more renames, rename the variable that points to the
final users table.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: as suggested by Arnout, use ROOTFS_FULL_USERS_TABLE instead
of ROOTFS_FINAL_USERS_TABLE.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 21:30:12 +01:00
Peter Korsgaard
13c43455a0 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-02 08:16:10 +01:00
Carlos Santos
21c6fb2589 fs/common: allow filesystems to set the name of their output file
Some filesystems may want to tweak their output names, rather than using
the fixed "rootfs.foo" scheme. Add a ROOTFS_FOO_IMAGE_NAME variable for
this purpose.

Signed-off-by: Carlos Santos <casantos@datacom.com.br>
[yann.morin.1998@free.fr: fix the patch]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-01 22:53:45 +01:00
Thomas Petazzoni
d0f4f95e39 Makefile: rework main directory creation logic
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.

For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.

This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.

Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.

Thanks to those changes, the global "dirs" target is entirely removed.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:09:46 +01:00
Yann E. MORIN
6d688e2132 fs/tar: add support for xattrs (thus capabilties)
By default, tar will not include any extended attribute (xattr) when
creating archives, and thus will not store capabilties either (as they
are stored in the xattr 'security.capability').

Using option --xattrs is enough to create a tarball with all the xattrs
attached to a file. However, extracting all xattrs from a tarball
requires that --xattrs-include='*' be used. This is not symetric (but on
purpose, as per the documentation), and so is confusing to some.

So, we use --xattrs-include='*' to create the archive, so as to be
explicit that we want all xattrs to be stored.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-20 23:28:07 +01:00
Serj Kalichev
305e4487e5 fs/common.mk: Fix show-build-order
The command "make show-build-order" doesn't show dependencies of rootfs-common target.

This patch adds $(ROOTFS_COMMON_DEPENDENCIES) to PACKAGES variable.

Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-18 09:46:52 +01:00
Yann E. MORIN
409d4c3fe9 fs: don't use an intermediate tarball
Since 118534fe54 (fs: use a common tarball as base for the other
filesystems), the filesystem creation is split in two steps, using an
intermediate tarball to carry the generic, common finalisations to the
per-filesystem finalisation and image creation.

However, this intermediate tarball causes an issue with capabilities:
they are entirely missing in the generated filesystems.

Capabilities are stored in the extended attribute security.capability,
which tar by default will not store/restore, unless explicitly told to,
e.g. with --xattrs-include='*', which we don't pass.

Now, passing this option when creating and extracting the intermediate
tarball, both done under fakeroot, will cause fakeroot to report an
invalid filetype for files with capabilities. mksquashfs would report
such unknown files as a warning, while mkfs.ext2 would fail (with a
similar error message), e.g.:

    File [...]/usr/sbin/getcap has unrecognised filetype 0, ignoring

This is due to a poor interaction between tar and fakeroot; running as
root the exact same commands we run under fakeroot, works as expected.
Unfortunately, short of fixing fakeroot (which would first require
understanding the problem in there), we don't have much options.

The intermediate tarball was made to avoid redoing the same actions over
and over again for each filesystem to build. However, most of the time,
only one or two such filesystems would be enabled [0], and those actions
are usually pretty lightweight. So, using an intermediate tarball does
not provide a big optimisation.

The main reason to introduce the intermediate tarball, however, is that
it allows to postpone per-filesystem finalisations to be applied only
for the corresponding filesystem, not for all of them.

So, we get rid of the intermediate tarball, and simply move all of the
code to run under fakeroot to the per-filesystem fakeroot script.
Instead of extracting the intermediate tarball, we just rsync the
original target/ directory, and apply the filesystem finalisations on
that copy. The only thing still done in the rootfs-common step is to
generate the intermediate files (users file, devices file) that are used
in the fakeroot script.

Fixes: https://bugs.busybox.net/show_bug.cgi?id=11216

Note: an alternate solution would have been to keep the intermediate
tarball to keep most of the common finalisations, and move only the
permissions to each filesystem, but that was getting a bit more complex
and changed the ordering of permissions and post-fakeroot scripts. Once
we bite the bullet of having some common finalisation done in each
filesystem, it's easier to just move all of them.

[0] Most probsably, users would enable the real filesystem to put on
their device, plus the 'tar' filesystem, to be able to easily inspect
the content on their development machine.

Reported-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-11-13 00:47:36 +01:00
Grzegorz Blach
3946f74b00 fs/f2fs: simplify overprovision option handling
Setting overprovision to 0 and omitting this option has exactly
the same effect.

Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-08 21:17:49 +01:00
Grzegorz Blach
25076c94c0 fs/f2fs: add option to define list of hot file extensions
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-08 21:07:40 +01:00
Grzegorz Blach
c3159a2d14 fs/f2fs: clarify help messages
For BR2_TARGET_ROOTFS_F2FS_COLD_FILES clarify that extension list must
be a coma separated.

For BR2_TARGET_ROOTFS_F2FS_OVERPROVISION clarify that the default
overprovision ratio is autocalculated according to the partition size.

Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-08 21:06:27 +01:00
Thomas Petazzoni
9617ce5cf9 fs/f2fs/Config.in: remove consecutive empty lines
This fixes the following check-package warning:

fs/f2fs/Config.in:51: consecutive empty lines

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:49:57 +01:00
Grzegorz Blach
3c66b119ed fs/f2fs: add option to define list of filesystem features
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
[Thomas: split from the initial patch from Grzegorz]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:32:52 +01:00
Grzegorz Blach
d4e5c0925a fs/f2fs: add option to define discard policy
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
[Thomas: split from the initial patch from Grzegorz]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:29:32 +01:00
Grzegorz Blach
0d27baa16a fs/f2fs: add option to define overprovision ratio
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
[Thomas: split from the initial patch from Grzegorz, reworded
Config.in help text]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:28:00 +01:00
Grzegorz Blach
0446f34d97 fs/f2fs: add option to define list of cold file extensions
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
[Thomas: split from the initial patch from Grzegorz]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:17:02 +01:00
Grzegorz Blach
6f727ce1fe fs/f2fs: add support for creating a f2fs image
This patch makes possible to create rootfs image using f2fs
filesystem.

Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
[Thomas:
 - keep only the minimal functionality, as suggested by Yann E. Morin
 - use truncate -s instead of dd to create the initial empty image
   file, as suggested by Yann E. Morin]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-03 15:14:47 +01:00
Yann E. MORIN
89c92487cf fs/common: always depend on build host-tar if needed
Currently, the filesystems do not depend on building host-tar when it
is needed, even though all of them have to extract the intermediate
tarball.

However, in degenerate (but legally valid) configurations with no
user-selectable package selected, host-tar would not be built, so the
rootfs images would use whatever improper tar the system has.

Add the conditional dependency to host-tar to the rootfs-common
intermediate image. Since this is the internal step that all real rootfs
generators depend on, they now properly depend on host-tar when needed.

In practice, when host-tar is needed, it will always be built before the
rootfs images, because it is a dependency of all packages (except a very
few, like the skeleton), of which host-fakeroot, which is a mandatory
dependency of rootfs-comon anyway. But for consistency sake, let's
explicitly add host-tar as a dependency to rootfs-common too.

Note that rootfs-tar already had that dependency, and we leave it as-is
because it is semantically correct, even if superfluous.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 22:01:46 +01:00
Matt Weber
c3edec0018 fs/common: allow custom user table to override package-defined users
Currently, when a custom user table and a package define the same user,
the settings from the package takes precedence over the ones from the
custom user table.

However, it makes sense to allow the settings from the custom user table
take precedence. For example, it would allow redirecting the user's
home directory to an alternate location (e.g. away from tmp and into a
partition that is persistent).

The support/scripts/mkusers script will only retain settings from the
latest definition it finds.

Thus, by passing the custom user table after the package defined users,
it is possible to override the package provided user definitions.

Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-10-20 14:33:24 +01:00
Yann E. MORIN
f023810f04 fs/btrfs: remove destination file before generation
btrfs will happily use an existing destination file if it
already exists, increasing its size if needed. Hoever, it
will never decrease the size, even if the requested size
is smaller than the existing file.

So, remove any previously existing destination file before
generating the new filesystem.

Note: the original submission by Robert did that, but as
this case was not obvious, the removal was dropped by a
refactoring when the patch was initially applied.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Robert J. Heywood <robert.heywood@codethink.co.uk>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Thomas: use $@ instead of $(@), use $(RM) instead of rm.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-25 23:59:23 +02:00
Robert J. Heywood
0aade4df1f fs/btrfs: add support for generating a btrfs image
This patch makes it possible to format the rootfs using btrfs. It
introduces the option; BR2_TARGET_ROOTFS_BTRFS.

When selected, the user is able to specify the filesystem size, label,
options, and node and sector sizes. The new files are based on
fs/ext2/{Config.in,ext2.mk}

Signed-off-by: Robert J. Heywood <robert.heywood@codethink.co.uk>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas:
 - fix issues pointed by Yann (duplicated empty line, missing quotes
   around default values for string options)
 - use -f option so that we don't have to remove the image file before
   creating it again
 - use the --byte-count option to set the filesystem size, which
   avoids the need for doing a "truncate -s"
 - remove the possible explanation of a mkfs.btrfs error. Indeed,
   mkfs.btrfs automatically extends the size of the image as needed,
   so the size passed can never be "too small".
 - fix check-package warnings in Config.in file.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-08-25 14:30:51 +02:00
Peter Korsgaard
065e7672e1 fs: ensure hard links in TARGET_DIR are correctly copied for filesystem input
Fixes 

Commit bb2a57a17a (fs: run packages' filesystem hooks in a copy of target/)
changed the file system logic to run file system hooks from packages on a
copy of TARGET_DIR, and finally use this copy as input for the file system
generation.

This copy was done with rsync, which by default does not preserve hard
links, leading to an expansion of the file system images when hard links are
present.

Fix it by passing the -H option to rsync (preserve hard links).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-05-27 23:46:29 +02:00
Peter Seiderer
124a149b7d fs/squashfs: add zstd support
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-25 23:39:04 +02:00
Chris Brandt
265aee8c51 fs/cramfs: add xip option
Cramfs now has the ability to XIP all ELF files.

Cc: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Chris Brandt <chris.brandt@renesas.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-09 11:29:46 +02:00
Chris Brandt
2cf4871808 package/cramfs: change to new site location
As of Linux-4.15, cramfs now has a official maintainer again.
Additionally, that person is hosting and maintaining a new version of
cramfs-tools.

Patches 0001-endian.patch and 0003-fix-missing-types.patch are no longer
needed because they have been upstreamed.

However, since patch 0002-cygwin_IO.patch is so old, it is being
removed until someone that needs it can reapply and test it. At
that point, they should submit the changes to the new cramfs-tools
maintainer instead of adding the patches back here.

Please note that cross-endian support for cramfsck is not supported
at this time.

Cc: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Chris Brandt <chris.brandt@renesas.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-09 11:29:43 +02:00
Ricardo Martincoski
4a417b7509 fs/*/Config.in: re-wrap help text
... to follow the convention <tab><2 spaces><62 chars>.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:52:45 +02:00
Ricardo Martincoski
b205420e6a fs/*/Config.in: fix attributes order
... to follow the convention: type, default, depends on, select, help.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:52:45 +02:00
Ricardo Martincoski
9b6b4e36b4 */Config.in*: remove consecutive empty lines
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:48:24 +02:00
Ricardo Martincoski
65e3743945 fs/*/*.mk: fix code style
The header of the initramfs.mk file fits in one line, so rearrange it.
Remove consecutive empty line.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:40:41 +02:00
Ricardo Martincoski
8f2b06f455 fs/ubi: use namespace for internal variable
UBINIZE_CONFIG_FILE_PATH is only used inside this file, so rename it to
start with UBI_, following the namespace convention already used by
common packages.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 08:40:29 +02:00
Ricardo Martincoski
0c90c109cb fs/yaffs2: fix code style
Rename the .mk to follow the convention. The variables inside it already
use the correct name.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-04-01 07:55:22 +02:00
Yann E. MORIN
543107d390 fs: remove intermediate artefacts
Each of the intermediate, per-rootfs target directories, as well as the
intermediate tarball, can take quite some place, and is mostly a
duplication of what's already in target/. The only delta, if any, would
be the tweaks made by the filesystem image generations, but those tweaks
are most probably only meaningful when seen as root.

We normally do not remove intermediate files, but those can be quite
large, and are not directly usable by, nor accessible to the user.
So, get rid of them once the filesystem has been generated.

This does not need to be done in fakeroot.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-31 20:53:06 +02:00
Yann E. MORIN
fcdf58cad1 fs: get rid of package-provided post-fs hooks
Now that the pre-fs ones are run on a transient copy of target/, the
post-fs hooks are no longer needed because we no longer need to restore
the target/ directory as it is only a internal copy.

Remove support for the post-fs hooks, and update the sole package using
them.

We do not add a legacy check because this was mostly a purely-internal
detail that was never really exposed nor documented.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-31 20:53:06 +02:00
Yann E. MORIN
bb2a57a17a fs: run packages' filesystem hooks in a copy of target/
Currently, some packages may register hooks to be run just before and
just after the generic tarball image is generated, because they need to
prepare the filesystem for read-only or read-write operation.

However, this means that, if any of the hooks or the image generation
fails, the target directory is left in a dangling, inconsistent state.

We fix that by doing a copy of target/, run the hooks on that copy,
generate the generic tarball image out of that, and get rid of the copy.

This way, we can guarantee consistency of the target directory, and we
can even ditch support for post-fs hooks (those that restore target/).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-31 20:53:06 +02:00
Yann E. MORIN
118534fe54 fs: use a common tarball as base for the other filesystems
Currently, some filesystems may want to tweak the content of the target
directory, create special device nodes etc... This all means that:

  - the content of the target directory for a specific filesystems may
    depend on whether another filesystem is enabled or not; for example,
    cpio will create a /init script or symlink and a /dev/console node;

  - the filesystems can not be built in parallel, because they may change
    the content of the target directory while another is being assembled.

Furthermore, the same fakeroot script is executed over-and-over-again
for each filesystem, to create the device nodes, the users and their
homes and files, and setting permissions...

We introduce an intermediate tarball, for which we do the full fakeroot
shebang.

That tarball then serves as the base for the other filesystems, with a
very simple fakeroot script that untars the common tarball, and calls
the actual filesystem image generator on that.

Note that we use a very simple tar command to generate the intermediate
tarball, because we are not concerned with reproducibility of the
archive itself (only of the archived files).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-03-31 20:53:06 +02:00