a9dc2de551
The current documentation was poorly organized, with for example the "Here is an example walk through of running a test case" sentence followed by the explanation of how to list available test cases, but not how to run one. Many other aspects of the wording were confusing, or not really accurate. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
713 lines
30 KiB
Plaintext
713 lines
30 KiB
Plaintext
// -*- mode:doc; -*-
|
|
// vim: set syntax=asciidoc:
|
|
|
|
== Contributing to Buildroot
|
|
|
|
There are many ways in which you can contribute to Buildroot: analyzing
|
|
and fixing bugs, analyzing and fixing package build failures detected by
|
|
the autobuilders, testing and reviewing patches sent by other
|
|
developers, working on the items in our TODO list and sending your own
|
|
improvements to Buildroot or its manual. The following sections give a
|
|
little more detail on each of these items.
|
|
|
|
If you are interested in contributing to Buildroot, the first thing you
|
|
should do is to subscribe to the Buildroot mailing list. This list is
|
|
the main way of interacting with other Buildroot developers and to send
|
|
contributions to. If you aren't subscribed yet, then refer to
|
|
xref:community-resources[] for the subscription link.
|
|
|
|
If you are going to touch the code, it is highly recommended to use a
|
|
git repository of Buildroot, rather than starting from an extracted
|
|
source code tarball. Git is the easiest way to develop from and directly
|
|
send your patches to the mailing list. Refer to xref:getting-buildroot[]
|
|
for more information on obtaining a Buildroot git tree.
|
|
|
|
=== Reproducing, analyzing and fixing bugs
|
|
|
|
A first way of contributing is to have a look at the open bug reports in
|
|
the https://bugs.buildroot.org/buglist.cgi?product=buildroot[Buildroot bug
|
|
tracker]. As we strive to keep the bug count as small as possible, all
|
|
help in reproducing, analyzing and fixing reported bugs is more than
|
|
welcome. Don't hesitate to add a comment to bug reports reporting your
|
|
findings, even if you don't yet see the full picture.
|
|
|
|
=== Analyzing and fixing autobuild failures
|
|
|
|
The Buildroot autobuilders are a set of build machines that continuously
|
|
run Buildroot builds based on random configurations. This is done for
|
|
all architectures supported by Buildroot, with various toolchains, and
|
|
with a random selection of packages. With the large commit activity on
|
|
Buildroot, these autobuilders are a great help in detecting problems
|
|
very early after commit.
|
|
|
|
All build results are available at http://autobuild.buildroot.org[],
|
|
statistics are at http://autobuild.buildroot.org/stats.php[]. Every day,
|
|
an overview of all failed packages is sent to the mailing list.
|
|
|
|
Detecting problems is great, but obviously these problems have to be
|
|
fixed as well. Your contribution is very welcome here! There are
|
|
basically two things that can be done:
|
|
|
|
- Analyzing the problems. The daily summary mails do not contain details
|
|
about the actual failures: in order to see what's going on you have to
|
|
open the build log and check the last output. Having someone doing
|
|
this for all packages in the mail is very useful for other developers,
|
|
as they can make a quick initial analysis based on this output alone.
|
|
|
|
- Fixing a problem. When fixing autobuild failures, you should follow
|
|
these steps:
|
|
. Check if you can reproduce the problem by building with the same
|
|
configuration. You can do this manually, or use the
|
|
http://git.buildroot.org/buildroot-test/tree/utils/br-reproduce-build[br-reproduce-build]
|
|
script that will automatically clone a Buildroot git repository,
|
|
checkout the correct revision, download and set the right
|
|
configuration, and start the build.
|
|
. Analyze the problem and create a fix.
|
|
. Verify that the problem is really fixed by starting from a clean
|
|
Buildroot tree and only applying your fix.
|
|
. Send the fix to the Buildroot mailing list (see
|
|
xref:submitting-patches[]). In case you created a patch against the
|
|
package sources, you should also send the patch upstream so that the
|
|
problem will be fixed in a later release, and the patch in Buildroot
|
|
can be removed.
|
|
In the commit message of a patch fixing an autobuild failure, add a
|
|
reference to the build result directory, as follows:
|
|
---------------------
|
|
Fixes: http://autobuild.buildroot.org/results/51000a9d4656afe9e0ea6f07b9f8ed374c2e4069
|
|
---------------------
|
|
|
|
=== Reviewing and testing patches
|
|
|
|
With the amount of patches sent to the mailing list each day, the
|
|
maintainer has a very hard job to judge which patches are ready to apply
|
|
and which ones aren't. Contributors can greatly help here by reviewing
|
|
and testing these patches.
|
|
|
|
In the review process, do not hesitate to respond to patch submissions
|
|
for remarks, suggestions or anything that will help everyone to
|
|
understand the patches and make them better. Please use internet
|
|
style replies in plain text emails when responding to patch
|
|
submissions.
|
|
|
|
To indicate approval of a patch, there are three formal tags that keep
|
|
track of this approval. To add your tag to a patch, reply to it with the
|
|
approval tag below the original author's Signed-off-by line. These tags
|
|
will be picked up automatically by patchwork (see
|
|
xref:apply-patches-patchwork[]) and will be part of the commit log when
|
|
the patch is accepted.
|
|
|
|
Tested-by:: Indicates that the patch has been tested successfully.
|
|
You are encouraged to specify what kind of testing you performed
|
|
(compile-test on architecture X and Y, runtime test on target A,
|
|
...). This additional information helps other testers and the
|
|
maintainer.
|
|
|
|
Reviewed-by:: Indicates that you code-reviewed the patch and did your
|
|
best in spotting problems, but you are not sufficiently familiar with
|
|
the area touched to provide an Acked-by tag. This means that there
|
|
may be remaining problems in the patch that would be spotted by
|
|
someone with more experience in that area. Should such problems be
|
|
detected, your Reviewed-by tag remains appropriate and you cannot
|
|
be blamed.
|
|
|
|
Acked-by:: Indicates that you code-reviewed the patch and you are
|
|
familiar enough with the area touched to feel that the patch can be
|
|
committed as-is (no additional changes required). In case it later
|
|
turns out that something is wrong with the patch, your Acked-by could
|
|
be considered inappropriate. The difference between Acked-by and
|
|
Reviewed-by is thus mainly that you are prepared to take the blame on
|
|
Acked patches, but not on Reviewed ones.
|
|
|
|
If you reviewed a patch and have comments on it, you should simply reply
|
|
to the patch stating these comments, without providing a Reviewed-by or
|
|
Acked-by tag. These tags should only be provided if you judge the patch
|
|
to be good as it is.
|
|
|
|
It is important to note that neither Reviewed-by nor Acked-by imply
|
|
that testing has been performed. To indicate that you both reviewed and
|
|
tested the patch, provide two separate tags (Reviewed/Acked-by and
|
|
Tested-by).
|
|
|
|
Note also that _any developer_ can provide Tested/Reviewed/Acked-by
|
|
tags, without exception, and we encourage everyone to do this. Buildroot
|
|
does not have a defined group of _core_ developers, it just so happens
|
|
that some developers are more active than others. The maintainer will
|
|
value tags according to the track record of their submitter. Tags
|
|
provided by a regular contributor will naturally be trusted more than
|
|
tags provided by a newcomer. As you provide tags more regularly, your
|
|
'trustworthiness' (in the eyes of the maintainer) will go up, but _any_
|
|
tag provided is valuable.
|
|
|
|
Buildroot's Patchwork website can be used to pull in patches for testing
|
|
purposes. Please see xref:apply-patches-patchwork[] for more
|
|
information on using Buildroot's Patchwork website to apply patches.
|
|
|
|
[[apply-patches-patchwork]]
|
|
==== Applying Patches from Patchwork
|
|
|
|
The main use of Buildroot's Patchwork website for a developer is for
|
|
pulling in patches into their local git repository for testing
|
|
purposes.
|
|
|
|
When browsing patches in the patchwork management interface, an +mbox+
|
|
link is provided at the top of the page. Copy this link address and
|
|
run the following commands:
|
|
|
|
---------------------
|
|
$ git checkout -b <test-branch-name>
|
|
$ wget -O - <mbox-url> | git am
|
|
---------------------
|
|
|
|
Another option for applying patches is to create a bundle. A bundle is
|
|
a set of patches that you can group together using the patchwork
|
|
interface. Once the bundle is created and the bundle is made public,
|
|
you can copy the +mbox+ link for the bundle and apply the bundle
|
|
using the above commands.
|
|
|
|
|
|
=== Work on items from the TODO list
|
|
|
|
If you want to contribute to Buildroot but don't know where to start,
|
|
and you don't like any of the above topics, you can always work on items
|
|
from the http://elinux.org/Buildroot#Todo_list[Buildroot TODO list].
|
|
Don't hesitate to discuss an item first on the mailing list or on IRC.
|
|
Do edit the wiki to indicate when you start working on an item, so we
|
|
avoid duplicate efforts.
|
|
|
|
[[submitting-patches]]
|
|
=== Submitting patches
|
|
|
|
[NOTE]
|
|
_Please, do not attach patches to bugs, send them to the mailing list
|
|
instead_.
|
|
|
|
If you made some changes to Buildroot and you would like to contribute
|
|
them to the Buildroot project, proceed as follows.
|
|
|
|
==== The formatting of a patch
|
|
|
|
We expect patches to be formatted in a specific way. This is necessary
|
|
to make it easy to review patches, to be able to apply them easily to
|
|
the git repository, to make it easy to find back in the history how
|
|
and why things have changed, and to make it possible to use +git
|
|
bisect+ to locate the origin of a problem.
|
|
|
|
First of all, it is essential that the patch has a good commit
|
|
message. The commit message should start with a separate line with a
|
|
brief summary of the change, prefixed by the area touched by the
|
|
patch. A few examples of good commit titles:
|
|
|
|
* +package/linuxptp: bump version to 2.0+
|
|
|
|
* +configs/imx23evk: bump Linux version to 4.19+
|
|
|
|
* +package/pkg-generic: postpone evaluation of dependency conditions+
|
|
|
|
* +boot/uboot: needs host-{flex,bison}+
|
|
|
|
* +support/testing: add python-ubjson tests+
|
|
|
|
The description that follows the prefix should start with a lower case
|
|
letter (i.e "bump", "needs", "postpone", "add" in the above examples).
|
|
|
|
Second, the body of the commit message should describe _why_ this
|
|
change is needed, and if necessary also give details about _how_ it
|
|
was done. When writing the commit message, think of how the reviewers
|
|
will read it, but also think about how you will read it when you look
|
|
at this change again a few years down the line.
|
|
|
|
Third, the patch itself should do only one change, but do it
|
|
completely. Two unrelated or weakly related changes should usually be
|
|
done in two separate patches. This usually means that a patch affects
|
|
only a single package. If several changes are related, it is often
|
|
still possible to split them up in small patches and apply them in a
|
|
specific order. Small patches make it easier to review, and often
|
|
make it easier to understand afterwards why a change was done.
|
|
However, each patch must be complete. It is not allowed that the
|
|
build is broken when only the first but not the second patch is
|
|
applied. This is necessary to be able to use +git bisect+ afterwards.
|
|
|
|
Of course, while you're doing your development, you're probably going
|
|
back and forth between packages, and certainly not committing things
|
|
immediately in a way that is clean enough for submission. So most
|
|
developers rewrite the history of commits to produce a clean set of
|
|
commits that is appropriate for submission. To do this, you need to
|
|
use _interactive rebasing_. You can learn about it
|
|
https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History[in the Pro
|
|
Git book]. Sometimes, it is even easier to discard you history with
|
|
+git reset --soft origin/master+ and select individual changes with
|
|
+git add -i+ or +git add -p+.
|
|
|
|
Finally, the patch should be signed off. This is done by adding
|
|
+Signed-off-by: Your Real Name <your@email.address>+ at the end of the
|
|
commit message. +git commit -s+ does that for you, if configured
|
|
properly. The +Signed-off-by+ tag means that you publish the patch
|
|
under the Buildroot license (i.e. GPL-2.0+, except for package patches,
|
|
which have the upstream license), and that you are allowed to do so.
|
|
See http://developercertificate.org/[the Developer Certificate of
|
|
Origin] for details.
|
|
|
|
When adding new packages, you should submit every package in a
|
|
separate patch. This patch should have the update to
|
|
+package/Config.in+, the package +Config.in+ file, the +.mk+ file, the
|
|
+.hash+ file, any init script, and all package patches. If the package
|
|
has many sub-options, these are sometimes better added as separate
|
|
follow-up patches. The summary line should be something like
|
|
+<packagename>: new package+. The body of the commit message can be
|
|
empty for simple packages, or it can contain the description of the
|
|
package (like the Config.in help text). If anything special has to be
|
|
done to build the package, this should also be explained explicitly in
|
|
the commit message body.
|
|
|
|
When you bump a package to a new version, you should also submit a
|
|
separate patch for each package. Don't forget to update the +.hash+
|
|
file, or add it if it doesn't exist yet. Also don't forget to check if
|
|
the +_LICENSE+ and +_LICENSE_FILES+ are still valid. The summary line
|
|
should be something like +<packagename>: bump to version <new
|
|
version>+. If the new version only contains security updates compared
|
|
to the existing one, the summary should be +<packagename>: security
|
|
bump to version <new version>+ and the commit message body should show
|
|
the CVE numbers that are fixed. If some package patches can be removed
|
|
in the new version, it should be explained explicitly why they can be
|
|
removed, preferably with the upstream commit ID. Also any other
|
|
required changes should be explained explicitly, like configure
|
|
options that no longer exist or are no longer needed.
|
|
|
|
If you are interested in getting notified of build failures and of
|
|
further changes in the packages you added or modified, please add
|
|
yourself to the DEVELOPERS file. This should be done in the same patch
|
|
creating or modifying the package. See xref:DEVELOPERS[the DEVELOPERS file]
|
|
for more information.
|
|
|
|
Buildroot provides a handy tool to check for common coding style
|
|
mistakes on files you created or modified, called +check-package+ (see
|
|
xref:check-package[] for more information).
|
|
|
|
==== Preparing a patch series
|
|
|
|
Starting from the changes committed in your local git view, _rebase_
|
|
your development branch on top of the upstream tree before generating
|
|
a patch set. To do so, run:
|
|
|
|
---------------------
|
|
$ git fetch --all --tags
|
|
$ git rebase origin/master
|
|
---------------------
|
|
|
|
Now, you are ready to generate then submit your patch set.
|
|
|
|
To generate it, run:
|
|
|
|
---------------------
|
|
$ git format-patch -M -n -s -o outgoing origin/master
|
|
---------------------
|
|
|
|
This will generate patch files in the +outgoing+ subdirectory,
|
|
automatically adding the +Signed-off-by+ line.
|
|
|
|
Once patch files are generated, you can review/edit the commit message
|
|
before submitting them, using your favorite text editor.
|
|
|
|
Buildroot provides a handy tool to know to whom your patches should be
|
|
sent, called +get-developers+ (see xref:DEVELOPERS[] for more
|
|
information). This tool reads your patches and outputs the appropriate
|
|
+git send-email+ command to use:
|
|
|
|
---------------------
|
|
$ ./utils/get-developers outgoing/*
|
|
---------------------
|
|
|
|
Use the output of +get-developers+ to send your patches:
|
|
|
|
---------------------
|
|
$ git send-email --to buildroot@buildroot.org --cc bob --cc alice outgoing/*
|
|
---------------------
|
|
|
|
Alternatively, +get-developers -e+ can be used directly with the
|
|
+--cc-cmd+ argument to +git send-email+ to automatically CC the
|
|
affected developers:
|
|
|
|
---------------------
|
|
$ git send-email --to buildroot@buildroot.org \
|
|
--cc-cmd './utils/get-developers -e' origin/master
|
|
---------------------
|
|
|
|
+git+ can be configured to automatically do this out of the box with:
|
|
|
|
---------------------
|
|
$ git config sendemail.to buildroot@buildroot.org
|
|
$ git config sendemail.ccCmd "$(pwd)/utils/get-developers -e"
|
|
---------------------
|
|
|
|
And then just do:
|
|
|
|
---------------------
|
|
$ git send-email origin/master
|
|
---------------------
|
|
|
|
Note that +git+ should be configured to use your mail account.
|
|
To configure +git+, see +man git-send-email+ or google it.
|
|
|
|
If you do not use +git send-email+, make sure posted *patches are not
|
|
line-wrapped*, otherwise they cannot easily be applied. In such a case,
|
|
fix your e-mail client, or better yet, learn to use +git send-email+.
|
|
|
|
==== Cover letter
|
|
|
|
If you want to present the whole patch set in a separate mail, add
|
|
+--cover-letter+ to the +git format-patch+ command (see +man
|
|
git-format-patch+ for further information). This will generate a
|
|
template for an introduction e-mail to your patch series.
|
|
|
|
A 'cover letter' may be useful to introduce the changes you propose
|
|
in the following cases:
|
|
|
|
* large number of commits in the series;
|
|
|
|
* deep impact of the changes in the rest of the project;
|
|
|
|
* RFC footnote:[RFC: (Request for comments) change proposal];
|
|
|
|
* whenever you feel it will help presenting your work, your choices,
|
|
the review process, etc.
|
|
|
|
==== Patches for maintenance branches
|
|
|
|
When fixing bugs on a maintenance branch, bugs should be fixed on the
|
|
master branch first. The commit log for such a patch may then contain a
|
|
post-commit note specifying what branches are affected:
|
|
|
|
----
|
|
package/foo: fix stuff
|
|
|
|
Signed-off-by: Your Real Name <your@email.address>
|
|
---
|
|
Backport to: 2020.02.x, 2020.05.x
|
|
(2020.08.x not affected as the version was bumped)
|
|
----
|
|
|
|
Those changes will then be backported by a maintainer to the affected
|
|
branches.
|
|
|
|
However, some bugs may apply only to a specific release, for example
|
|
because it is using an older version of a package. In that case, patches
|
|
should be based off the maintenance branch, and the patch subject prefix
|
|
must include the maintenance branch name (for example "[PATCH 2020.02.x]").
|
|
This can be done with the +git format-patch+ flag +--subject-prefix+:
|
|
|
|
---------------------
|
|
$ git format-patch --subject-prefix "PATCH 2020.02.x" \
|
|
-M -s -o outgoing origin/2020.02.x
|
|
---------------------
|
|
|
|
Then send the patches with +git send-email+, as described above.
|
|
|
|
==== Patch revision changelog
|
|
|
|
When improvements are requested, the new revision of each commit
|
|
should include a changelog of the modifications between each
|
|
submission. Note that when your patch series is introduced by a cover
|
|
letter, an overall changelog may be added to the cover letter in
|
|
addition to the changelog in the individual commits.
|
|
The best thing to rework a patch series is by interactive rebasing:
|
|
+git rebase -i origin/master+. Consult the git manual for more
|
|
information.
|
|
|
|
When added to the individual commits, this changelog is added when
|
|
editing the commit message. Below the +Signed-off-by+ section, add
|
|
+---+ and your changelog.
|
|
|
|
Although the changelog will be visible for the reviewers in the mail
|
|
thread, as well as in http://patchwork.buildroot.org[patchwork], +git+
|
|
will automatically ignores lines below +---+ when the patch will be
|
|
merged. This is the intended behavior: the changelog is not meant to
|
|
be preserved forever in the +git+ history of the project.
|
|
|
|
Hereafter the recommended layout:
|
|
|
|
---------------
|
|
Patch title: short explanation, max 72 chars
|
|
|
|
A paragraph that explains the problem, and how it manifests itself. If
|
|
the problem is complex, it is OK to add more paragraphs. All paragraphs
|
|
should be wrapped at 72 characters.
|
|
|
|
A paragraph that explains the root cause of the problem. Again, more
|
|
than one paragraph is OK.
|
|
|
|
Finally, one or more paragraphs that explain how the problem is solved.
|
|
Don't hesitate to explain complex solutions in detail.
|
|
|
|
Signed-off-by: John DOE <john.doe@example.net>
|
|
|
|
---
|
|
Changes v2 -> v3:
|
|
- foo bar (suggested by Jane)
|
|
- bar buz
|
|
|
|
Changes v1 -> v2:
|
|
- alpha bravo (suggested by John)
|
|
- charly delta
|
|
---------------
|
|
|
|
Any patch revision should include the version number. The version number
|
|
is simply composed of the letter +v+ followed by an +integer+ greater or
|
|
equal to two (i.e. "PATCH v2", "PATCH v3" ...).
|
|
|
|
This can be easily handled with +git format-patch+ by using the option
|
|
+--subject-prefix+:
|
|
|
|
---------------------
|
|
$ git format-patch --subject-prefix "PATCH v4" \
|
|
-M -s -o outgoing origin/master
|
|
---------------------
|
|
|
|
Since git version 1.8.1, you can also use +-v <n>+ (where <n> is the
|
|
version number):
|
|
|
|
---------------------
|
|
$ git format-patch -v4 -M -s -o outgoing origin/master
|
|
---------------------
|
|
|
|
When you provide a new version of a patch, please mark the old one as
|
|
superseded in http://patchwork.buildroot.org[patchwork]. You need to
|
|
create an account on http://patchwork.buildroot.org[patchwork] to be
|
|
able to modify the status of your patches. Note that you can only change
|
|
the status of patches you submitted yourself, which means the email
|
|
address you register in http://patchwork.buildroot.org[patchwork] should
|
|
match the one you use for sending patches to the mailing list.
|
|
|
|
You can also add the +--in-reply-to <message-id>+ option when
|
|
submitting a patch to the mailing list. The id of the mail to reply to
|
|
can be found under the "Message Id" tag on
|
|
http://patchwork.buildroot.org[patchwork]. The advantage of
|
|
*in-reply-to* is that patchwork will automatically mark the previous
|
|
version of the patch as superseded.
|
|
|
|
[[reporting-bugs]]
|
|
=== Reporting issues/bugs or getting help
|
|
|
|
Before reporting any issue, please check in
|
|
xref:community-resources[the mailing list archive] whether someone has
|
|
already reported and/or fixed a similar problem.
|
|
|
|
However you choose to report bugs or get help, either by
|
|
opening a bug in the xref:community-resources[bug tracker] or by
|
|
xref:community-resources[sending a mail to the mailing list], there are
|
|
a number of details to provide in order to help people reproduce and
|
|
find a solution to the issue.
|
|
|
|
Try to think as if you were trying to help someone else; in
|
|
that case, what would you need?
|
|
|
|
Here is a short list of details to provide in such case:
|
|
|
|
* host machine (OS/release)
|
|
* version of Buildroot
|
|
* target for which the build fails
|
|
* package(s) for which the build fails
|
|
* the command that fails and its output
|
|
* any information you think that may be relevant
|
|
|
|
Additionally, you should add the +.config+ file (or if you know how, a
|
|
+defconfig+; see xref:customize-store-buildroot-config[]).
|
|
|
|
If some of these details are too large, do not hesitate to use a
|
|
pastebin service. Note that not all available pastebin services will
|
|
preserve Unix-style line terminators when downloading raw pastes.
|
|
Following pastebin services are known to work correctly:
|
|
- https://gist.github.com/
|
|
- http://code.bulix.org/
|
|
|
|
=== Using the runtime tests framework
|
|
|
|
Buildroot includes a run-time testing framework built upon Python
|
|
scripting and QEMU runtime execution. The goals of the framework are
|
|
the following:
|
|
|
|
* build a well defined Buildroot configuration
|
|
* optionally, verify some properties of the build output
|
|
* optionally, boot the build results under Qemu, and verify that a
|
|
given feature is working as expected
|
|
|
|
The entry point to use the runtime tests framework is the
|
|
+support/testing/run-tests+ tool, which has a series of options
|
|
documented in the tool's help '-h' description. Some common options
|
|
include setting the download folder, the output folder, keeping build
|
|
output, and for multiple test cases, you can set the JLEVEL for each.
|
|
|
|
Here is an example walk through of running a test case.
|
|
|
|
* For a first step, let us see what all the test case options are. The test
|
|
cases can be listed by executing +support/testing/run-tests -l+. These tests
|
|
can all be run individually during test development from the console. Both
|
|
one at a time and selectively as a group of a subset of tests.
|
|
|
|
---------------------
|
|
$ support/testing/run-tests -l
|
|
List of tests
|
|
test_run (tests.utils.test_check_package.TestCheckPackage)
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootMusl) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainCCache) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainCtngMusl) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainLinaroArm) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv4) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv5) ... ok
|
|
test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv7) ... ok
|
|
[snip]
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRoFull) ... ok
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRoIfupdown) ... ok
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRoNetworkd) ... ok
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRwFull) ... ok
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRwIfupdown) ... ok
|
|
test_run (tests.init.test_systemd.TestInitSystemSystemdRwNetworkd) ... ok
|
|
test_run (tests.init.test_busybox.TestInitSystemBusyboxRo) ... ok
|
|
test_run (tests.init.test_busybox.TestInitSystemBusyboxRoNet) ... ok
|
|
test_run (tests.init.test_busybox.TestInitSystemBusyboxRw) ... ok
|
|
test_run (tests.init.test_busybox.TestInitSystemBusyboxRwNet) ... ok
|
|
|
|
Ran 157 tests in 0.021s
|
|
|
|
OK
|
|
---------------------
|
|
|
|
* Then, to run one test case:
|
|
|
|
---------------------
|
|
$ support/testing/run-tests -d dl -o output_folder -k tests.init.test_busybox.TestInitSystemBusyboxRw
|
|
15:03:26 TestInitSystemBusyboxRw Starting
|
|
15:03:28 TestInitSystemBusyboxRw Building
|
|
15:08:18 TestInitSystemBusyboxRw Building done
|
|
15:08:27 TestInitSystemBusyboxRw Cleaning up
|
|
.
|
|
Ran 1 test in 301.140s
|
|
|
|
OK
|
|
---------------------
|
|
|
|
The standard output indicates if the test is successful or not. By
|
|
default, the output folder for the test is deleted automatically
|
|
unless the option +-k+ is passed to *keep* the output directory.
|
|
|
|
==== Creating a test case
|
|
|
|
Within the Buildroot repository, the testing framework is organized at the
|
|
top level in +support/testing/+ by folders of +conf+, +infra+ and +tests+.
|
|
All the test cases live under the +tests+ folder and are organized in various
|
|
folders representing the category of test.
|
|
|
|
The best way to get familiar with how to create a test case is to look
|
|
at a few of the basic file system +support/testing/tests/fs/+ and init
|
|
+support/testing/tests/init/+ test scripts. Those tests give good
|
|
examples of a basic tests that include both checking the build
|
|
results, and doing runtime tests. There are other more advanced cases
|
|
that use things like nested +br2-external+ folders to provide
|
|
skeletons and additional packages.
|
|
|
|
Creating a basic test case involves:
|
|
|
|
* Defining a test class that inherits from +infra.basetest.BRTest+
|
|
|
|
* Defining the +config+ member of the test class, to the Buildroot
|
|
configuration to build for this test case. It can optionally rely on
|
|
configuration snippets provided by the runtime test infrastructure:
|
|
+infra.basetest.BASIC_TOOLCHAIN_CONFIG+ to get a basic
|
|
architecture/toolchain configuration, and
|
|
+infra.basetest.MINIMAL_CONFIG+ to not build any filesystem. The
|
|
advantage of using +infra.basetest.BASIC_TOOLCHAIN_CONFIG+ is that a
|
|
matching Linux kernel image is provided, which allows to boot the
|
|
resulting image in Qemu without having to build a Linux kernel image
|
|
as part of the test case, therefore significant decreasing the build
|
|
time required for the test case.
|
|
|
|
* Implementing a +def test_run(self):+ function to implement the
|
|
actual tests to run after the build has completed. They may be tests
|
|
that verify the build output, by running command on the host using
|
|
the +run_cmd_on_host()+ helper function. Or they may boot the
|
|
generated system in Qemu using the +Emulator+ object available as
|
|
+self.emulator+ in the test case. For example +self.emulator.boot()+
|
|
allows to boot the system in Qemu, +self.emulator.login()+ allows to
|
|
login, +self.emulator.run()+ allows to run shell commands inside
|
|
Qemu.
|
|
|
|
After creating the test script, add yourself to the +DEVELOPERS+ file to
|
|
be the maintainer of that test case.
|
|
|
|
==== Debugging a test case
|
|
|
|
When a test case runs, the +output_folder+ will contain the following:
|
|
|
|
---------------------
|
|
$ ls output_folder/
|
|
TestInitSystemBusyboxRw/
|
|
TestInitSystemBusyboxRw-build.log
|
|
TestInitSystemBusyboxRw-run.log
|
|
---------------------
|
|
|
|
+TestInitSystemBusyboxRw/+ is the Buildroot output directory, and it
|
|
is preserved only if the +-k+ option is passed.
|
|
|
|
+TestInitSystemBusyboxRw-build.log+ is the log of the Buildroot build.
|
|
|
|
+TestInitSystemBusyboxRw-run.log+ is the log of the Qemu boot and
|
|
test. This file will only exist if the build was successful and the
|
|
test case involves booting under Qemu.
|
|
|
|
If you want to manually run Qemu to do manual tests of the build
|
|
result, the first few lines of +TestInitSystemBusyboxRw-run.log+
|
|
contain the Qemu command line to use.
|
|
|
|
You can also make modifications to the current sources inside the
|
|
+output_folder+ (e.g. for debug purposes) and rerun the standard
|
|
Buildroot make targets (in order to regenerate the complete image with
|
|
the new modifications) and then rerun the test.
|
|
|
|
==== Runtime tests and Gitlab CI
|
|
|
|
All runtime tests are regularly executed by Buildroot Gitlab CI
|
|
infrastructure, see .gitlab.yml and
|
|
https://gitlab.com/buildroot.org/buildroot/-/jobs.
|
|
|
|
You can also use Gitlab CI to test your new test cases, or verify that
|
|
existing tests continue to work after making changes in Buildroot.
|
|
|
|
In order to achieve this, you need to create a fork of the Buildroot
|
|
project on Gitlab, and be able to push branches to your Buildroot fork
|
|
on Gitlab.
|
|
|
|
The name of the branch that you push will determine if a Gitlab CI
|
|
pipeline will be triggered or not, and for which test cases.
|
|
|
|
In the examples below, the <name> component of the branch name is an
|
|
arbitrary string you choose.
|
|
|
|
* To trigger all run-test test case jobs, push a branch that ends with
|
|
+-runtime-tests+:
|
|
|
|
---------------------
|
|
$ git push gitlab HEAD:<name>-runtime-tests
|
|
---------------------
|
|
|
|
* To trigger one or several test case jobs, push a branch that ends
|
|
with the complete test case name
|
|
(+tests.init.test_busybox.TestInitSystemBusyboxRo+) or with the name
|
|
of a category of tests (+tests.init.test_busybox+):
|
|
|
|
---------------------
|
|
$ git push gitlab HEAD:<name>-<test case name>
|
|
---------------------
|
|
|
|
Example to run one test:
|
|
|
|
---------------------
|
|
$ git push gitlab HEAD:foo-tests.init.test_busybox.TestInitSystemBusyboxRo
|
|
---------------------
|
|
|
|
Examples to run several tests part of the same group:
|
|
|
|
---------------------
|
|
$ git push gitlab HEAD:foo-tests.init.test_busybox
|
|
$ git push gitlab HEAD:foo-tests.init
|
|
---------------------
|