Commit Graph

567 Commits

Author SHA1 Message Date
Thomas Petazzoni
b102352b62 support/scripts/pkg-stats: add --disable option
When debugging pkg-stats, it's quite useful to be able to disable some
features that are quite long (checking upstream URL, checking latest
version, checking CVE). This commit adds a --disable option, which can
take a comma-separated list of features to disable, such as:

     ./support/scripts/pkg-stats --disable url,upstream

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-01-09 17:31:55 +01:00
Gregory CLEMENT
a206bbc5fe support/scripts/pkg-stats: account for unsure CVEs
The .affects() method of the CVE class in support/scripts/cve.py can
return 3 values: CVE_AFFECTS, CVE_DOESNT_AFFECT and CVE_UNKNOWN.

We of course properly account for CVEs where .affects() return
CVE_AFFECTS, but the ones for which CVE_UNKNOWN is returned are
currently ignored, and therefore treated as if they did not affect the
package.

However CVE_UNKNOWN in fact indicates that the v_start/v_end fields of
the CPE entry could not be parsed by
distutils.version.LooseVersion(). Instead of ignoring such cases, this
commit adds support for the concept of "unsure CVEs", which will be
listed next to CVEs known to affect the package, so that we are aware
of them and can investigate the version issue.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-01-09 17:31:55 +01:00
Romain Naour
b1cc6061fd support/scripts/generate-gitlab-ci-yml: ignore branch name prefix <foo> containing a single hyphen
The commit [1] added a sed command used to retreive a pattern
to keep only defconfigs whose name start with the pattern.

"<foo>-defconfigs-<pattern>"

The sed command doesn't work as expected if <foo> contains a
single hyphen [2]:

"qemu-6.2.0-defconfigs-qemu"

Update the sed command to ignore completely the part before
"-defconfigs-".

[1] 65d2f04c01
[2] http://lists.busybox.net/pipermail/buildroot/2022-January/632507.html

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-08 15:48:48 +01:00
Andrey Nechypurenko
1368ce2331 support/scripts/: remove -E flag from patch call
-E flag instructs patch to remove empty files. However, in some cases
empty files are essential. If they are missing, build could be broken
or other bad things can happen.

Note that empty files are still removed when their headers are properly
formattedo: timestamp set to 1970-01-01T00:00:00Z, destination set to
/dev/null.

Signed-off-by: Andrey Nechypurenko <andreynech@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-01-03 21:14:32 +01:00
Thomas Petazzoni
bbfcb19c55 support/scripts/gen-bootlin-toolchains: handle RISC-V 64-bit toolchain change
toolchains.bootlin.com no longer provides a LP64 RISC-V 64-bit
toolchain, but a more useful LP64D RISC-V 64-bit toolchain. Of course,
the old tarballs remain available, but no new versions of the LP64
toolchain will be produced.

This commit reflects this change in the gen-bootlin-toolchains script.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-12-30 18:43:14 +01:00
Thomas Petazzoni
65d2f04c01 support/scripts/generate-gitlab-ci-yml: make it possible to test multiple defconfigs
Already supported:

 - Pushing a branch called "<foo>-defconfigs" tests all defconfigs.

 - Pushing a branch called "<foo>-defconfig-<defconfig-name>" will
   test one particular defconfig

This commit adds support for:

 - Pushing a branch called "<foo>-defconfigs-<pattern>" which will
   test all defconfigs whose name start with the pattern. For example
   "<foo>-defconfigs-qemu_" will test all Qemu defconfigs

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-12-30 18:41:53 +01:00
Cyril Bur
c988867fd2 support/scripts/pkg-stats: Don't print directory in 'make' call
It is possible that some users of buildroot have put it in a repository
and call into it from another Makefile such as:
.DEFAULT:
	$(MAKE) O=$(abspath $(O)) -C buildroot $(@)

This technique works well except that Make tells us that it changes into
the buildroot directory:
make[1]: Entering directory 'buildroot'

Because this line doesn't have an equals within it, python raises a
ValueError exception within pkg-stats.

This patch has python tell the invoked make not to print directories

Signed-off-by: Cyril Bur <cyrilbur@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-12-11 21:35:19 +01:00
Yann E. MORIN
b919d5dbba support/pkg-stats: support values with an equal sign in them
The heuristic to extract the various variables of interest is pretty
crude: we filter on variables ending with certain suffixes (like
'%_VERSION' to get the version strings).

However, in doing so, we may dump variables that are not actual package
versions (especially with br2-external trees), and those may contain one
or more equal sign.  And anyway, an actual package version string may
very well contain an equal sign too.

But the current situation is that the output of 'printvars' is split on
all equal signs, which will not fit in the 2-tuple we assign the result,
thus causing an exception.

Fix that by limiting to a single split.

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-13 14:59:42 +01:00
Romain Naour
8ea6eead60 support/scripts: don't build board defconfigs with Gitlab's pipelines trigged on tag
Currently when a tag is added to the Buildroot git tree, the gitlab-ci
create a pipeline with several hundred of jobs (~750) to build all
defconfigs and execute the Buildroot testsuite.

However, there is only a limited number of gitlab-ci runner (9 runners)
and some jobs reach the timeout limit (24h) while waiting for a runner
[1]. Indeed, the Buildroot project doesn't use the Gitlab's shared
runners.

In addition to the pipeline created when a new tag is added to the
git repository, two pipelines are created each weeks to execute the
Buildroot testsuite (on monday [2]) and build all defconfigs (on
Thursday [3]).

At some point there are too many jobs waiting in gitlab due board
defconfigs builds. Indded a board defconfig requires a lot of time
(~30min) compared to other jobs in order to build a toolchain and a
kernel linux along with a basic rootfs. There is currently 262
defconfigs.

This is even worse when several pipelines are trigged at the same
time (new git tag and scheduled pipeline trigger).

In order to reduce the number of long jobs, don't build board
defconfigs with pipelines trigged on tag, keeping only the runtime
tests and the Qemu's defconfigs.

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966541
[2] https://gitlab.com/buildroot.org/buildroot/-/pipelines/404035190
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/401685550

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-11-11 09:24:47 +01:00
Thomas Petazzoni
23186356a1 support/scripts/generate-gitlab-ci-yml: allow multiple tests in one pipeline
The current Gitlab CI mechanism allows to trigger all tests in a CI
pipeline by pushing a branch named <something>-runtime-tests, or to
trigger a single test in a CI pipeline by pushing a branch name
<something>-tests.<name of test>.

However, there are cases where it is useful to run a suite of tests,
for example to run all tests in tests.init.test_busybox.

This commit makes that possible by extending the current semantic of
<something>-tests.<name of test> to not expect a complete test name,
but instead to accept all tests that starts with the given pattern.

This allows to do:

  git push gitlab HEAD:foobar-tests.init.test_busybox.TestInitSystemBusyboxRo

like it was the case before. But it now also allows to do:

  git push gitlab HEAD:foobar-tests.init.test_busybox

to run all Busybox tests.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-10-27 21:31:49 +02:00
Thomas Petazzoni
11efcb39b2 support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org
The pkg-stats script queries release-monitoring.org to find the latest
upstream versions of our packages. However, up until recently,
release-monitoring.org had no notion of stable
vs. development/release-candidate versions, so for some packages the
"latest" version was in fact a development/release-candidate version
that we didn't want to package in Buildroot.

However, in recent time, release-monitoring.org has gained support for
differentiating stable vs. development releases of upstream
projects. See for example
https://release-monitoring.org/project/10024/ for the glib library,
which has a number of versions marked "Pre-release".

The JSON blurb returned by release-monitoring.org has 3 relevant
fields:

 - "version", which we are using currently, which is a string
   containing the reference of the latest version, including
   pre-release.

 - "versions", which is an array of strings listing all versions,
   pre-release or not.

 - "stable_versions", which is an array of string listing only
   non-pre-release versions. It is ordered newest first to oldest
   last.

So, this commit changes from using 'version' to using
'stable_versions[0]'.

As an example, before this change, pkg-stats reports that nfs-utils
needs to be bumped to 2.5.5rc3, while after this patch, it reports
that nfs-utils is already at 2.5.4, and that this is the latest stable
version (modulo an issue where Buildroot has 2.5.4 and
release-monitoring.org has 2-5-4, this will be addressed separately).

Note that part of this change was already done in commit f7b0e0860, but
it was incomplete.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-10-21 21:13:00 +02:00
Thomas Petazzoni
9602fd94e7 support/scripts/pkg-stats: fix the status for packages found by guess
The pkg-stats scripts tries to match packages against
release-monitoring.org in two ways:

- First by using the "Buildroot" distribution registered on
  release-monitoring.org, in which we have added a lot of mappings
  between Buildroot package names and release-monitoring.org package
  names. If there is a match using this distribution, the package
  status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
  resulting HTML has a "found by distro" statement.

- Then, if the first solution didn't work, by using the pattern
  matching, as done in the check_package_get_latest_version_by_guess()
  function.

However, there is a bug in this later case: it sets the package status
to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
file from a pkg-stats run, all packages are marked as "found by
distro" even the ones that are "found by guess".

This commit fixes that by setting the correct package status.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-10-21 21:04:58 +02:00
Francois Perrad
f7b0e08605 support/scripts/pkg-stats: prefers stable version from release-monitoring.org
For example with libpng: 1.6.37 instead of 1.7.0beta89

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
[yann.morin.1998@free.fr: coalesce into a single line]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-10-19 21:57:22 +02:00
Arnout Vandecappelle (Essensium/Mind)
d721c95b8b support: fix flake8 error E741 ambiguous variable name
A recent update of flake8 in CI introduced a new check E741. It
basically checks that variables are at least 3 characters long. Up to
now, however, we have used shorter names in some places - all of them
turn out to be "l" for a line of text.

Replace all those "l" variables with "line".

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/1687009829
partially:
support/scripts/boot-qemu-image.py:47:21: E741 ambiguous variable name 'l'
support/scripts/check-dotconfig.py:20:38: E741 ambiguous variable name 'l'
support/scripts/size-stats:76:13: E741 ambiguous variable name 'l'
support/testing/tests/core/test_bad_arch.py:17:32: E741 ambiguous variable name 'l'
support/testing/tests/package/test_python_treq.py:10:30: E741 ambiguous variable name 'l'
support/testing/tests/toolchain/test_external.py:30:42: E741 ambiguous variable name 'l'

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-10-18 22:31:11 +02:00
Petr Vorel
35f15db30a support: utils: use python3 explicitly
Python 2 is EOL sice 2020 [1], it's still available on distros, but may not
be installed by default (as being replaced by python3).

Thus remove compatibility imports:
from __future__ import print_function
from __future__ import absolute_import

Tested with python3 -m py_compile.

[1] https://www.python.org/doc/sunset-python-2/

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-09-22 21:30:24 +02:00
Romain Naour
12c7a05da1 utils/test-pkg: add gitlab-ci support
The gitlab-ci support in test-pkg allows to parallelize the test-pkg
work into several gitlab jobs. It's much faster than local serialized
testing.

To trigger this, a developer will have to add, in the latest commit of
their branch, a token on its own line, followed by a configuration
fragment, e.g.:

    test-pkg config:
    SOME_OPTION=y
    # OTHER_OPTION is not set
    SOME_VARIABLE="some value"

This configuration fragment is used as input to test-pkg.

To be able to generate one job per test to run, we need the list of
tests in the parent pipeline, and the individual .config files (one per
test) in the child pipeline. We use the newly-introduced --prepare-only
mode to test-pkg, and collect all the generated .config files as
artefacts; those are inherited in the child pipeline via the
"needs::pipeline" and "needs::job" directives. This is a bit tricky,
and is best described by the Gitlab-CI documentation [0].

We also list those .config files to generate the actual list of jobs to
run in the child pipeline.

Notes:
  - if the user provides an empty fragment, this is considered an error:
    indeed, without a fragment (and the package name), there is no way
    to know what to test;
  - if that fragment yields an empty list of tests, then there is
    nothing to test either, so that is also considered an error.

[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998@free.fr:
  - split the change to test-pkg to its own patch
  - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
    listing the .config files created by test-pkg
  - some code-style-candies...
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-08-28 17:03:13 +02:00
Peter Korsgaard
8d07baab43 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-06-07 17:14:37 +02:00
Matthew Weber
50791af71f support/scripts/pkg-stats: clarify when a CVE/CPE should report as N/A
- If a package doesn't have any versioning, ignore and state that
 - If a package is virtual, CVE=ignore and CPE state virtual
 - For any of these NA cases, don't provide search link and color box
   green

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-19 10:21:28 +02:00
Matthew Weber
9368f3f23d support/scripts/pkg-stats: add is_actual_package() and rework has_valid_infra()
has_valid_infra() is incorrectly named; it probably should be named
is_actual_package(), and has_valid_infra() would be changed to
actually represent having an actual infra.

This resolves packages reporting as having no valid package infra and
cleans up reporting cases of CPE and CVEs where there isn't a valid version
or package definition outside Buildroot

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-19 10:14:31 +02:00
Matthew Weber
9b83bb1382 support/scripts/pkg-stats: verified CPE has a known id but not version
Currently a verified CPE reports the following if versions are not found
 cpe:2.3🅰️qemu:qemu:5.2.0:*:*:*:*:*:*:*
 CPE identifier unknown in CPE database (Search)

This patch clarifies the report to state the 'version' is unknown instead
of the 'identifier'.

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-19 09:26:50 +02:00
Ryota Kinukawa
f46e13f05f support/dependencies, scripts: accept patches with renames
Currently, patches with renames are refused, as they reqire patch 2.7
or newer. So far, we did not require that version because it was too
recent to be widely available.

But patch 2.7 has been released in 2012, almost 9 years ago now; it is
old enough that we can start relying on it.

Add a check that patch is GNU patch 2.7 or newer, and so drop the common
check for patch, and drop the check about renames in apply-patches.sh.

Signed-off-by: Ryota Kinukawa <pojiro.jp@gmail.com>
[yann.morin.1998@free.fr:
  - drop common check
  - shorten variable names
  - drop now-incorrect comment about busybox w/desktop
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-19 08:33:11 +02:00
Matthew Weber
c2ebfe7d78 support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-18 21:06:12 +02:00
Matt Weber
f3b07e2ded support/scripts/pkg-stats: add column reporting ignored CVEs
When doing analysis it is helpful to be able to view what CVE have
been patched / diagnosed to not apply to Buildroot. This exposes
that list to the reporting and prevents a step where you have to
dig into the .mk's of a pkg to check for sure what has been
ignored.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[yann.morin.1998@free.fr: only set background if there are ignored CVEs]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-16 18:22:58 +02:00
Matt Weber
c5aa3c5883 support/scripts/pkg-stats: add CPE searching links
For cases of a CPE having a unknown version or when there hasn't
been a CPE verified, proposed a search criteria to help the
user research an update.

(libcurl has NIST dict entries but not this version)
  cpe:2.3🅰️haxx:libcurl:7.76.1:*:*:*:*:*:*:*
  CPE identifier unknown in CPE database (Search)

(jitterentropy-library package doesn't have any NIST dict entries)
  no verified CPE identifier (Search)

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[yann.morin.1998@free.fr: fix flake8 issues]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-16 17:54:25 +02:00
Matt Weber
fffc553485 support/scripts/gen-missing-cpe: add new script
This script queries the list of CPE IDs for the packages of the
current configuration (based on the "make show-info" output), and:

 - for CPE IDs that do not have any matching entry in the CPE
   database, it emits a warning

 - for CPE IDs that do have a matching entry, but not with the same
   version, it generates a snippet of XML that can be used to propose
   an updated version to NIST.

Ref: NIST has a group email (cpe_dictionary@nist.gov) used to
recieve these version update and new entry xml files.  They do
process the XML and provide feedback. In some cases they will
propose back something different where the vendor or version is
slightly different.

Limitations
 - Currently any use of non-number version identifiers isn't
   supported by NIST as they use ranges to determine impact
   of a CVE
 - Any Linux version from a non-upstream is also not supported
   without manually adjusting the information as the custom
   kernel will more then likely not match the upstream version
   used in the dictionary

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
[yann.morin.1998@free.fr:
  - codestyles as spotted by Arnout
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-05-16 13:57:38 +02:00
Thomas Petazzoni
d06bf96097 support/scripts/cve.py: use proper CPE ID version when available
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-04-18 18:20:27 +02:00
Peter Korsgaard
f31227e628 support/scripts/cve.py: use fast ijson backend if available on old ijson versions
ijson < 2.5 (as available in Debian 10) use the slow python backend by
default instead of the most efficient one available like modern ijson
versions, significantly slowing down cve checking. E.G.:

time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html

Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total

To
93,53s user 2,00s system 98% cpu 1:36,65 total

E.G. almost 2x as fast.

As a workaround, detect when the python backend is used and try to use a
more efficient one instead.  Use the yajl2_cffi backend as recommended by
upstream, as it is most likely to work, and print a warning (and continue)
if we fail to load it.

The detection is slightly complicated by the fact that ijson.backends used
to be a reference to a backend module, but is nowadays a string (without the
ijson.backends prefix).

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-04-17 09:14:40 +02:00
Bert Outtier
6fdc878d73 support/scripts: fix pycompile for short filenames
Signed-off-by: Bert Outtier <outtierbert@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-03-30 22:18:44 +02:00
Peter Korsgaard
a0aff89ed2 support/scripts/gen-bootlin-toolchains: correct xtensa-lx60 toolchain dependencies
Fixes:
http://autobuild.buildroot.net/results/011/0111c2ed54618daaeedfc66b0ea04eda00a7e855/
http://autobuild.buildroot.net/results/e53/e53e3880b63a23fa3b3e6d34664d40d5ddbdff89/
..

As listed in the br_fragment file of the toolchain, this is built for a
little-endian "custom" xtensa variant rather than the (big-endian) fsf one:

BR2_xtensa=y
BR2_XTENSA_CUSTOM=y

So update the dependencies in the script and regenerate Config.in.options /
toolchain test.  Also fixup the autobuild config snippet to match.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-03-02 23:52:49 +01:00
Romain Naour
03c3fbd81c support/scripts/boot-qemu-image.py: properly catch timeout
As reported on IRC by sephthir, the gitlab test of the defconfig
qemu_sparc_ss10_defconfig doesn't error out while the system
is not working properly.

This is because we explicitly wait for the timeout as an expected
condition, but do not check for it. Indeed, pexpect.expect() returns
the index of the matching condition in the list of expected conditions,
but we just ignore the return code, so we are not able to differentiate
between a successful login (or prompt) from a timeout.

By default, pexepect.expect() raises the pexpect.TIMEOUT exception on a
timeout, and we are already prepared to catch and handle that exception.
But because pexpect.TIMEOUT is passed as an expected condition, the
exception is not raised.

Remove pexpect.TIMEOUT from the list of expected conditions, so that the
exception is properly raised again, and so that we can catch it.

The qemu_sparc_ss10_defconfig is already fixed by
4d16e6f532.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
[yann.morin.1998@free.fr: reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-20 22:44:29 +01:00
Heiko Thiery
5b844d50d8 support/scripts/pkg-stats: add ignored_cves to json output
Add the list of <pkg>_IGNORE_CVES to the json output to show that we have a
known cause (available patch or the CVE is not valid for our package
configuration) that a affected CVE is not reported.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-02-16 23:15:06 +01:00
Arnout Vandecappelle (Essensium/Mind)
bcef4a03e7 support/scripts/cpedb.py: remove import pickle
pickle is no longer used since 09a71e6a75

Fixes:
support/scripts/cpedb.py:7:1: F401 'pickle' imported but unused

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-02-16 21:16:55 +01:00
Thomas Petazzoni
09a71e6a75 support/scripts/cpedb.py: drop CPE XML database caching
Currently, the CPE XML database is parsed into a Python dict, which is
then pickled into a local file, to speed up the processing of further
invocations.

However, it turns out that since the initial implementation, we have
switched the XML parsing from the out of tree xmltodict module to the
standard ElementTree one, which has made the parsing much faster. The
pickle caching only saves 6 seconds, on something that takes more than
13 minutes total.

In addition, this pickle caching consumes a significant amount of RAM,
causing the Python process to be OOM-killed on a server with 4 GB of
RAM.

So let's just drop this caching entirely.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-02-14 10:05:00 +01:00
Thomas Petazzoni
201e74bf9d support/scripts/pkg-stats: check CPE existence in CPE dictionnary
This commit extends pkg-stats to leverage the recently introduced
CPEDB class to verify that the CPEs provided by Buildroot packages are
indeed known in the official CPE dictionnary provided by NVD.

Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-02-02 20:16:59 +01:00
Matt Weber
ba8e4767d0 support/scripts/cpedb.py: new CPE XML helper
Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.

 - Defines the CPE as a object with operations / formats
 - Processing of CPE dictionary

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Co-Developed-by: Grégory Clement <gregory.clement@bootlin.com>
Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-02-02 20:16:49 +01:00
Peter Korsgaard
97f1d5d151 support/scripts/pkg-stats: get_config_packages(): use dict.values()
There is no need to get both the key and the value out of the dict if the
key is not used, so use dict.values() instead.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2021-02-02 16:34:10 +01:00
Thomas Petazzoni
c9a1a1eb91 support/scripts/pkg-stats: properly handle host packages with -c option
In commit 7a607dab33
("support/scripts/pkg-stats: support generating stats based on
configured packages"), we added a -c option to pkg-stats to generate a
report based on the list of packages enabled in the configuration,
rather than for all packages.

This is done based on the list of packages returned in JSON format by
"make show-info". However, we use the keys of the JSON dict returned
by "make show-info", which include the host- prefix of host
packages. Due to this, none of the host packages are currently
matching and therefore they are not reported in the pkg-stats -c
output.

This commit fixes that by using the recently introduced "name"
property in the "make show-info" JSON dict.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use anonymous '_' for unused variable]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2021-01-31 23:01:21 +01:00
Thomas Petazzoni
52a53c97e5 support/scripts/pkg-stats: fix flake8 warnings
Fixes:

support/scripts/pkg-stats:148:17: E741 ambiguous variable name 'l'
support/scripts/pkg-stats:379:9: E741 ambiguous variable name 'l'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:12 +01:00
Thomas Petazzoni
67768e97b0 support/scripts/pkg-stats: drop unused --cpeid option
The --cpeid option was mistakenly introduced by commit
92e7089a8c ("support/script/pkg-stats:
show CPE ID in results") but is in fact not necessary.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:11 +01:00
Thomas Petazzoni
cad5a69f83 support/scripts/pkg-stats: drop unused cpeid_name() function
The cpeid_name() function is not used anywhere, drop it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:03:00 +01:00
Thomas Petazzoni
68364a4b58 support/scripts/pkg-stats: fix the status reporting of CVEs
Since commit bd665d182c
("support/scripts/pkg-stats: improve rendering of CVE information"),
we have better reporting of CVE related information, based on
pkg.status['cve']. However, this commit broke pkg-stats when the
--nvd-path option is not passed, and therefore no CVE information is
available.

This commit fixes that, by making use of the is_status_ok(),
is_status_error() and is_status_na() methods recently introduced.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:02:42 +01:00
Thomas Petazzoni
7f83ad7467 support/scripts/pkg-stats: improvements in is_status_*() methods
Make is_status_ok() work when the given status name is not even listed
in the status dict. This will be necessary for following commits.

Introduced similar methods for the error and na status, which will be
used in following commits.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-31 12:02:41 +01:00
Thomas Petazzoni
40bb37bd70 utils/getdeveloperlib.py: use relative paths for files
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.

parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.

This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.

Before this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]

$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": []
}

After this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
  "Floris Bos <bos@je-eigen-domein.nl>",
  "Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": [
    "Christophe Priouzeau <christophe.priouzeau@st.com>"
  ]
}

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:08 +01:00
Thomas Petazzoni
ffb2620405 support/scripts/pkg-stats: fix Python 3.8 deprecation warning
With Python 3.8, the following deprecation warnings are emitted:

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:418: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:536: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

The correct way to pass coroutines is to use asyncio.create_task(),
but this is rather new method (Python 3.7), and using it breaks
compatibility with older Python versions. As suggested at
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task,
use the more cryptic, but also more compatible asyncio.ensure_future()
method.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:05 +01:00
Thomas Petazzoni
824032d168 support/scripts/pkg-stats: import cve module only when needed
The cve module needs ijson, which may not be installed. Since cve
matching is only enabled when --nvd-path is passed, it is a bit silly
to error out about ijson being missing if it's not used.

So instead of unconditionally importing the cve module, only do it
conditionally.

However, instead of doing it right at the point where it is used, we
do it at the beginning of the main() function. Indeed, if the cve
module is needed but cannot be imported, we want to error out
immediately rather than doing a whole bunch of things, and failing on
the user later on in the middle of the pkg-stats execution.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:04 +01:00
Arnout Vandecappelle (Essensium/Mind)
24dc403be3 support/scripts/pkg-stats: fix flake8 errors
support/scripts/pkg-stats:81:22: E211 whitespace before '('
support/scripts/pkg-stats:404:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/pkg-stats:561:12: E713 test for membership should be 'not in'
support/scripts/pkg-stats:567:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:595:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1051:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1057:1: E302 expected 2 blank lines, found 1

Also fix:
support/scripts/pkg-stats:1054:5: E722 do not use bare 'except'
found by a more recent flake8 version. The exception may be either
IndexError or AttributeError, so use Exception to catch either.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-01-05 23:34:28 +01:00
Thomas Petazzoni
bd665d182c support/scripts/pkg-stats: improve rendering of CVE information
This commit improves pkg-stats to fill in pkg.status['cve'] depending
on the situation for CVEs affecting this package. They are then used
in the HTML rendering.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:39:11 +01:00
Thomas Petazzoni
78d7521f82 support/scripts/pkg-stats: ignore packages with no valid infra and no version for CVE checking
Virtual packages (with in pkg-stats speak have "no valid
infrastructure") and packages that have no version specified cannot be
used for CVE checking. They trigger a bunch of warnings from the CVE
checking code, as it cannot parse their version: they don't have any
version. So instead, we simply skip those packages.

A follow-up commit will improve the reporting to be able to
distinguish those packages from packages that have seen their CVEs
checked and don't have any reported.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:38:35 +01:00
Thomas Petazzoni
e3ef352ef6 support/scripts/{pkg-stats, cve.py}: support CPE ID based matching
This commit modifies cve.py, as well as its users cve-checker and
pkg-stats to support CPE ID based matching, for packages that have CPE
ID information.

One of the non-trivial thing is that we can't simply iterate over all
CVEs, and then iterate over all our packages to see which packages
have CPE ID information that match the CPEs affected by the
CVE. Indeed, this is an O(n^2) operation.

So instead, we do a pre-filtering of packages potentially affected. In
check_package_cves(), we build a cpe_product_pkgs dict that associates
a CPE product name to the packages that have this CPE product
name. The CPE product name is either derived from the CPE information
provided by the package if available, and otherwise we use the package
name, which is what was used prior to this patch.

And then, when we look at CVEs, we only consider the packages that
have a CPE product name matching the CPE products affected by the
CVEs. This is done in check_package_cve_affects().

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:38:20 +01:00
Gregory CLEMENT
92e7089a8c support/script/pkg-stats: show CPE ID in results
This commit improves the pkg-stats script to show the CPE ID of
packages, if available. For now, it doesn't use CPE IDs to match CVEs.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:36:50 +01:00
Thomas Petazzoni
8cba5463de support/scripts/gen-bootlin-toolchains: do not assume RPC for glibc toolchains
The logic in gen-bootlin-toolchains was assuming all glibc toolchains
have RPC support, which is no longer true since glibc 2.32 has dropped
RPC support.

It turns out that gen-bootlin-toolchains already had some proper logic
that selects BR2_TOOLCHAIN_HAS_NATIVE_RPC depending on the presence of
BR2_TOOLCHAIN_EXTERNAL_INET_RPC in the toolchain fragment. As such
toolchain fragments have been fixed in https://toolchains.bootlin.com,
we can now rely on this to properly decide if the toolchain has RPC
support or not.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-29 22:03:17 +01:00
Romain Naour
d673618342 support/scripts/boot-qemu-image.py: don't fail if start-qemu.sh is missing
When boot-qemu-image.py script was added, we wanted to run
each qemu defconfig in gitlab, so we expect that all qemu
defconfig generate the script start-qemu.sh in images
directory.

Don't make it a hard requirement even if we prefer to be
able to do a runtime test for each qemu defconfig.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-12-05 22:33:27 +01:00
Thomas Petazzoni
1ff7f003e1 support/scripts/cve.py: properly match CPEs with version '*'
Currently, when the version encoded in a CPE is '-', we assume all
versions are affected, but when it's '*' with no further range
information, we assume no version is affected.

This doesn't make sense, so instead, we handle '*' and '-' in the same
way. If there's no version information available in the CVE CPE ID, we
assume all versions are affected.

This increases quite a bit the number of CVEs and package affected:

-    "total-cves": 302,
-    "pkg-cves": 100,
+    "total-cves": 597,
+    "pkg-cves": 135,

For example, CVE-2007-4476 has a CPE ID of:

    cpe:2.3🅰️gnu:tar:*:*:*:*:*:*:*:*

So it should be taken into account. In this specific case, it is
combined with an AND with CPE ID
cpe:2.3suse:suse_linux:10:*:enterprise_server:*:*:*:*:* but since
we don't support this kind of matching, we'd better be on the safe
side, and report this CVE as affecting tar, do an analysis of the CVE
impact, and document it in TAR_IGNORE_CVES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-11-26 16:31:37 +01:00
Thomas Petazzoni
3de9337781 support/scripts/cve-checker: remove script
Now that pkg-stats is able to generate its output based on the list of
packages enabled in the current configuration, cve-checker doesn't
serve any purpose.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-11 12:07:37 +01:00
Thomas Petazzoni
7a607dab33 support/scripts/pkg-stats: support generating stats based on configured packages
pkg-stats was initially a Buildroot maintenance oriented tool: it was
designed to examine all Buildroot packages and provide
statistics/details about them.

However, it turns out that a number of details provided by pkg-stats,
especially CVEs, are relevant also for Buildroot users, who would like
to check regularly if their specific Buildroot configuration is
affected by CVEs or not, and possibly check if all packages have
license information, license files, etc.

The cve-checker script was recently introduced to provide an output
relatively similar to pkg-stats, but focused on CVEs only.

But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses
"make show-info" to only consider packages enabled in the current
configuration.

So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
to generate its output based on the list of configured packages. -c is
mutually exclusive with the -p option (explicit list of packages) and
-n option (a number of packages, picked randomly).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-11 12:03:55 +01:00
Thomas Petazzoni
ae86067a15 support/scripts/pkg-stats: allow to run script outside of the top-level directory
Currently, pkg-stats expects being executed from Buildroot's top-level
source directory. As we are going to extend pkg-stats to cover only
the packages available in the current configuration, it makes sense to
be able to run it from the output directory, which can be anywhere
compared to Buildroot's top-level directory.

This commit adjusts pkg-stats to this, by inferring all Buildroot
paths based on the location of the pkg-stats script itself.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-11 12:01:21 +01:00
Fabrice Fontaine
35c3a1e693 support/scripts/apply-patches.sh: manage uncompression
Extract from bug report:

"Code line 120 to line 128 is to check whether the patch containing
"rename from" and "rename to". But it directly use grep to find,
ignoring the patch may be a tar file or else. It can only work on patch
of textfile form."

Fixes:
 - https://bugs.buildroot.org/show_bug.cgi?id=11931

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-17 14:44:05 +02:00
Thomas Petazzoni
1f3e5e6b3d support/scripts/gen-bootlin-toolchains: add support for more PowerPC toolchains
The 2020.08-1 release of Bootlin toolchains has brought support for 3
additional architecture variants, so let's support them.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-14 23:57:54 +02:00
Romain Naour
4ba1ce24b9 support/scripts/gen-bootlin-toolchains: add missing MMU dependencies
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-14 23:56:17 +02:00
Romain Naour
85d15bd200 support/scripts/gen-bootlin-toolchains: add missing glibc toolchain dependencies
glibc toolchains must be disabled for static only configuration.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-14 23:55:56 +02:00
Brandon Maier
a0b041d6c1 system: support br2-external init systems
Some externals may wish to provide custom init systems for tightly
integrated boot. This has been supported through the BR2_INIT_NONE,
however a downside to the BR2_INIT_NONE is it forces the custom init
system to use either skeleton-custom and roll a custom skeleton for
each target, or skeleton-init-none which isn't a complete skeleton.

Allowing br2-external to define custom BR2_INIT_* means they can now
safely 'select' the BR2_PACKAGE_SKELETON_INIT_*, and re-use any of the
skeletons in Buildroot, or one from a br2-external tree.

Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-10-14 22:48:42 +02:00
Brandon Maier
8a06daa1a0 system: support br2-external skeleton packages
Today, the BR2_ROOTFS_SKELETON_CUSTOM is the only way to build a custom
skeleton. But it's limiting as users must provide a pre-built skeleton
for each target. Supporting a br2-external package allows users to build
up a skeleton and customize it with their own KConfig options.

Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-10-14 22:30:56 +02:00
Peter Korsgaard
42f61e759a support/scripts/apply-patches.sh: do not blindly remove *.orig files
apply-patches currently blindly removes *.orig / .*.orig files as GNU patch
by default writes these as backup files when patches only apply with fuzz.

This is unfortunate as package sources may contain files ending in .orig as
well, breaking the build.  Luckily GNU patch can be told to not write these
backup files using the --no-backup-if-mismatch option, so used that instead
of the .orig removal step.

--no-backup-if-mismatch is supported since GNU patch 2.3.8 (1997-06-17) and
busybox patch if built with CONFIG_DESKTOP, but E.G.  isn't supported by the
BSD patch, so add logic to dependencies.sh to error out if patch doesn't
support the flag.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-10-06 23:40:25 +02:00
Yann E. MORIN
cf8364f202 support/scripts/generate-gitlab-ci-yml: rework generation of pipelines
Currently, we handle three kinds of tests: basic, defconfig, and
runtime, and we treat them totally independently ones from the others.

Except for the basic tests that are ignored when defconfig or runtime
tests are explicitly requested.

The basic tests are also run systematically on all our reference
branches: master, next (when it exists), and the maintenance branches:
YYYY.MM.x.

Furthermore, we can see that the conditions to run each set of tests
are very similar, with only the explicit queries differing by name.

Rework the script so that the conditions are expressed only once, and
each set of tests is decided for each condition. This makes it easier
to decide what tests should run under what conditions.

Using GitLab-CI's schedules, with a variable expressing the actual test
to run, would seem the obvious choice to trigger the pipelines. However,
a schedule is configured for a specific branch, which means we would
need one schedule per branch we want to build per test cases we want to
run, *and* that we update those schedules when we add/remove branches
(e.g. when we open/close 'next', or a maintenance branch). This is not
very nice, as it requires some manual tweaking and twiddling on the web
UI.

Instead, we resort to using triggers, that will be triggered from a
cronjob on some server. Using a cronjiob allows us to more easily manage
the branches we want to test and test cases we want to run, to more
easily spread the load over the week, etc...

Note: triggering a pipeline can be done with a simple curl invocation:

    $ curl -X POST \
        -F "token=${YOUR_TOKEN}" \
        -F "ref=${BRANCH_TO_TEST}" \
        -F "variables[BR_SCHEDULE_JOBS]=${TEST_TO_RUN}" \
        "https://gitlab.com/api/v4/projects/${YOUR_PROJECT_ID}/trigger/pipeline"

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-10-06 15:38:48 +02:00
Thomas De Schampheleire
32eb5a1d16 support/scripts/setlocalversion: fix/improve Mercurial output
Commit 9e4ffdc8cf modified the output of
'setlocalversion' so that the Buildroot version tag is included in the
output, the version part was added in Makefile.

Due to differences in behavior of the used git and Mercurial commands, this
caused different output for the Mercurial case, in BR2_VERSION_FULL and thus
/etc/os-release and 'make print-version'. Assuming the official Buildroot
releases are tagged and no project-specific tags are present, the output
after commit 9e4ffdc8cf is:
    -hg<commit>
whereas it is expected to be something like:
    2020.02.6-hg<commit>

Change the Mercurial case in setlocalversion to behave similar to git,
looking up the latest tag if the current revision is not itself tagged.

The number of commits after the latest tag is not added, unlike in git, as
this value is not commonly present in Mercurial output, and its added value
can be disputed in this context. Even one commit could bring a huge change
to the sources, so in order to interpret the number one has to look at the
repository anyhow, in which case the commit ID can just be used.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-10-03 09:11:03 +02:00
Yann E. MORIN
245a64c387 support/scripts: prioritize conditions for pipeline creation
When multiple conditions match simultaneously, even though that should
not happen in practice, we want the more "important" one to win over
the less "important" ones. For example, a tag is more important than a
branch name or a trigger.

Currently, the latest condition to match takes precendence over any
previous one, while we want the exact opposite.

Fix that with proper fallbacks in else-blocks.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-09-22 21:29:39 +02:00
Robin Jarry
f88dcd3f99 support/scripts/pycompile: add --verbose option
Add a new option that prints the (runtime) path of compiled .py files
when VERBOSE=1 is set.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-13 10:28:36 +02:00
Robin Jarry
c566f5206a support/scripts/pycompile: fix .pyc original source file paths
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.

By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.

This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.

Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.

Rework the script to call py_compile.compile() directly with pertinent
options:

- The script now has a new --strip-root argument. This argument is
  optional but will always be specified when compiling py files in
  buildroot.
- All other (non-optional) arguments are folders in which all
  "importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
  file is computed and encoded into the compiled .pyc.

No need to change directory before running the script anymore.

The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.

Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).

Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.

So we just drop support for --force, and instead we always byte-compile.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
  - always byte-compile
  - drop --force
  - expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-13 10:19:13 +02:00
Robin Jarry
4c77dca550 support/scripts/pycompile: sort imports
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 21:20:40 +02:00
Robin Jarry
7b3025f93e support/scripts/pycompile: add main entry point
Only run code when the script is executed directly (not imported).
Factorize command description by using the script's __doc__ variable.
Fix typo in --force help message.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 21:16:20 +02:00
Yann E. MORIN
8fb49636d6 gitlab-ci: only check defconfigs for known branches
Currently, the check of defconfigs is run for all branches, even those
that are pushed only to run runtime tests. This is very inconvenient.

In fact, we only want to check the defconfigs on standard branches, that
is master, next, and the maintenance branches.

This will also decrease drastically the number gitlab-ci minutes used
when one pushes their repo to gitlab.com, where the number of CI minutes
are now going to be pretty severely restricted.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:26:00 +02:00
Yann E. MORIN
6bc09ae38f gitlab-ci: handle the basic tests in script
Note that those tests were so far ignored only when requesting a single
defconfig build, or a single runtime test build; everything else
was trigerring thoses tests.

However, it feels more natural that they are also ignored when all
defconfigs build. or all runtime tests, are explictly requested.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:25:52 +02:00
Yann E. MORIN
0b7873bc82 gitlab-ci: handle single runtime test in script
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:25:51 +02:00
Yann E. MORIN
386f2c620b gitlab-ci: handle single defconfig in script
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:25:49 +02:00
Yann E. MORIN
f307d88774 gitlab-ci: handle the run-time tests conditions in script
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:24:58 +02:00
Yann E. MORIN
bc8ee069d2 gitlab-ci: defconfig_check is included in defconfig build
When we build the defconfigs, we already check they are correct, so
there is no need to run the correctness check explicitly.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Acked-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:24:20 +02:00
Yann E. MORIN
bca0f1bb41 gitlab-ci: handle the defconfig build conditions in script
Note that we do not propagate the existing comment, because it is
partially wrong; instead we just keep the per-condition comments.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:24:18 +02:00
Yann E. MORIN
bef6739094 gitlab-ci: share the image version with the child
Currently, the image name and version are duplicated in the main
pipeline and the generated, child pipeline.

This is a condition for a future gaffe, so let's use the image from the
main pipeline when generating the child one.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:24:17 +02:00
Yann E. MORIN
f70a0e9d91 gitlab-ci: introduce main() in generating script
This script is currently very crude, but we're going to extend it, at
which point it will be nicer to have functions, local variables, et al.

Introduce a main() in preparation of those future evolutions.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-09 09:24:16 +02:00
Yann E. MORIN
843334bf4f support/scripts: fix flake8 errors in cve-checker and pkg-stats
Note that one is silenced, rather than fixed: we indeed need to import
after we add the local directory to the modules search path.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-09-03 20:48:12 +02:00
Bernd Kuhls
6b2355fef0 package/nvidia-tegra23: remove package
Current X.org X server is incompatible with this driver.

We no longer support unmaintainted versions of X.org X server.

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2020-09-01 21:57:21 +02:00
Yann E. MORIN
e1bf9bedd5 support/scripts/gen-bootlin-toolchains: be less verbose
Drop the debug-level print as noticed by Titouan.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Titouan Christophe <titouan.christophe@railnova.eu>
2020-08-29 16:10:02 +02:00
Thomas Petazzoni
184d47a7ad support/scripts/gen-bootlin-toolchains: add new script to support Bootlin toolchains
https://toolchains.bootlin.com/ has been providing for a few years a
number of ready-to-use pre-built toolchains, for a wide range of
architectures (which it turns out, are all built using Buildroot).

While toolchains.bootlin.com provides Buildroot config fragments to
easily use those toolchains with Buildroot (see [0] for example), this
is not visible anywhere. So instead, we would like to add support for
these toolchains in Buildroot just like we have existing support for
Linaro, ARM, Synopsys, etc. toolchains.

[0] https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--bleeding-edge-2020.02-2.frag

However, the number of toolchains provided by toolchains.bootlin.com
is really large, and they are regularly updated. Maintaining that
manually would be time consuming and error-prone. So instead, this
commit introduces a script that automatically generates:

 - toolchain/toolchain-external/toolchain-external-bootlin/Config.in.options
 - toolchain/toolchain-external/toolchain-external-bootlin/toolchain-external-bootlin.mk
 - toolchain/toolchain-external/toolchain-external-bootlin/toolchain-external-bootlin.hash
 - support/testing/tests/toolchain/test_external_bootlin.py

We create a single external toolchain package, with a Kconfig "choice"
as a sub-option to select the toolchain variant to be used. The script
contains a Python dict that provides the mapping between the
toolchains provided by toolchains.bootlin.com, and the architecture
options/variants they are applicable to.

The test cases allow to verify that the toolchain configuration is
correct, and that it is able to build a Busybox based system. It
doesn't do any runtime testing as such testing is already done by
toolchains.bootlin.com: the test cases here are only meant to verify
that the toolchain-external-bootlin package works as expected.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-08-29 15:47:39 +02:00
Gregory CLEMENT
fafa3e4e29 support/scripts/cve-checker: add a per configuration CVE checker
This scripts takes as entry on stdin a JSON description of the package
used for a given configuration. This description is the one generated
by "make show-info".

The script generates the list of all the packages used and if they are
affected by a CVE. The output is either a JSON or an HTML file similar
to the one generated by pkg-stats.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>=
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 11:35:03 +02:00
Gregory CLEMENT
2a2f69d672 support/scripts: make CVE class independent of the Packaage class
The affects method of the CVE uses the Package class defined in
pkg-stats. The purpose of migrating the CVE class outside of pkg-stats
was to be able to reuse it from other scripts. So let's remove the
Package dependency and only use the needed information.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 11:00:40 +02:00
Gregory CLEMENT
52ae092046 support/scripts/cve.py: use the JSON data in 1.1 schema
In 2019, the JSON vulnerability feeds switched their schema from
version 1.0 to 1.1.

The main difference is the removal of the "affects" element that we
were using to check if a package was affected by a CVE.

This information is now available in the "configuration" element which
contains the cpeid as well as properties about the versions
affected. Instead of having a list of the versions affected, with
these properties, it is possible to have a range of versions.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:39 +02:00
Gregory CLEMENT
f8ee65570a support/scripts/pkg-stats: move CVE related code to a separate module
In order to be able to use the CVE checking logic outside of
pkg-stats, move the CVE class in a module that can be used by other
scripts.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
008ca2c583 support/scripts/pkg-stats: consider "-" as a wildcard when doing CVE version matching
Some CVE entries in the NVD database have version_value set to "-",
which seems to indicate that it applies to all versions of the
software project, or that they don't really know which versions are
affected, and which are not.

So, for the benefit of doubt, it seems more appropriate to consider
such CVEs as affecting our packages.

This makes the total number of CVEs affecting our next branch jump
from 141 CVEs to 658 CVEs, but that number will go back down once we
switch to the JSON 1.1 schema. Indeed, in the JSON 1.0 schema, there
are often cases where a version_value is set to "=" *and* specific
versions are set to.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
e631c314b1 support/scripts/pkg-stats: drop erroneous "break" in CVE.affects()
Commit 7d2779ecbb
("support/script/pkg-stats: handle exception when version comparison
fails") erroneousy introduced a "break" within a try/expect block.

This break has the unfortunate consequence that every CVE that was
using the <= operator was skipped, and according to the current
CVE statistics, made us miss 74 CVEs out of 141 CVEs.

Here is for reference the complete list of CVEs we missed:

 - gnupg
   CVE-2006-3082
   CVE-2019-13050

 - jhead
   CVE-2020-6624
   CVE-2020-6625

 - patch
   CVE-2018-6952
   CVE-2019-20633

 - json-c
   CVE-2020-12762

 - git
   CVE-2018-1000110
   CVE-2018-1000182
   CVE-2019-1003010
   CVE-2020-2136

 - iperf2
   CVE-2016-4303

 - libtorrent
   CVE-2009-1760
   CVE-2016-5301

 - lua
   CVE-2020-15888
   CVE-2020-15889
   CVE-2020-15945
   CVE-2020-24342

 - openvpn
   CVE-2020-7224

 - smack
   CVE-2016-10027

 - bashtop
   CVE-2019-18276

 - links
   CVE-2008-3319

 - argus
   CVE-2011-3332

 - libraw
   CVE-2020-15503

 - netcat
   CVE-2008-5727
   CVE-2008-5728
   CVE-2008-5729
   CVE-2008-5730
   CVE-2008-5742
   CVE-2015-2214

 - subversion
   CVE-2017-1000085
   CVE-2018-1000111
   CVE-2020-2111

 - python
   CVE-2013-1753
   CVE-2015-5652
   CVE-2017-17522
   CVE-2017-18207
   CVE-2019-20907
   CVE-2019-9674

 - cereal
   CVE-2020-11104
   CVE-2020-11105

 - opencv
   CVE-2017-1000450
   CVE-2017-12597
   CVE-2017-12598
   CVE-2017-12599
   CVE-2017-12600
   CVE-2017-12601
   CVE-2017-12602
   CVE-2017-12603
   CVE-2017-12604
   CVE-2017-12605
   CVE-2017-12606
   CVE-2017-12862
   CVE-2017-12863
   CVE-2017-12864
   CVE-2019-15939

 - docker
   CVE-2015-1843
   CVE-2015-3627
   CVE-2015-3630
   CVE-2015-3631
   CVE-2016-3697
   CVE-2017-14992
   CVE-2019-16884

 - trousers
   CVE-2020-24330
   CVE-2020-24331
   CVE-2020-24332

 - libcroco
   CVE-2020-12825

 - libpupnp
   CVE-2020-13848

 - openjpeg
   CVE-2020-15389

 - flex
   CVE-2015-1773

 - libesmtp
   CVE-2019-19977

 - ed
   CVE-2015-2987

 - libmad
   CVE-2018-7263

 - grub
   CVE-2020-15705

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
4f0868fa64 support/scripts/pkg-stats: show progress of upstream URL and latest version
This commit slightly improves the output of pkg-stats by showing the
progress of the upstream URL checks and latest version retrieval, on a
package basis:

Checking URL status
[0001/0062] curlpp
[0002/0062] cmocka
[0003/0062] snappy
[0004/0062] nload
[...]
[0060/0062] librtas
[0061/0062] libsilk
[0062/0062] jhead
Getting latest versions ...
[0001/0064] libglob
[0002/0064] perl-http-daemon
[0003/0064] shadowsocks-libev
[...]
[0061/0064] lua-flu
[0062/0064] python-aiohttp-security
[0063/0064] ljlinenoise
[0064/0064] matchbox-lib

Note that the above sample was run on 64 packages. Only 62 packages
appear for the URL status check, because packages that do not have any
URL in their Config.in file, or don't have any Config.in file at all,
are not checked and therefore not accounted.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:52 +02:00
Thomas Petazzoni
c8ba6aa133 support/scripts/pkg-stats: use aiohttp for upstream URL checking
This commit reworks the code that checks if the upstream URL of each
package (specified by its Config.in file) using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:43 +02:00
Thomas Petazzoni
d28b2bc481 support/scripts/pkg-stats: use aiohttp for latest version retrieval
This commit reworks the code that retrieves the latest upstream
version of each package from release-monitoring.org using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Since we're now using some async functionality, the script is Python
3.x only, so the shebang is changed to make this clear.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:25 +02:00
James Hilliard
84b9f07b87 package/guile: bump to version 3.0.4
Remove patch that is no longer needed as of upstream commit
1c33be992e8120abd20add8021e4d91d226f5b6a which removed the old VM.

We need to add an exclusion rule for guile modules to check-bin-arch
as they appear as valid ELF binaries but with an architecture of
"None".

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Thomas:
 - bump to 3.0.4
 - rework how check-bin-arch excludes checking the Guile .go files]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-08 18:59:19 +02:00
Thomas Petazzoni
98c99556e3 Makefile: properly account for custom tags in BR2_VERSION_FULL
BR2_VERSION_FULL is currently defined as follows:

  BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)

This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.

The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.

If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).

This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.

The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.

Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:

$ make print-version
2020.08-git-00758-gc351877a6e

So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:

$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e

This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.

So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.

In the above situation, with this commit applied, we get:

$ make print-version
dummy-tag-6-g6258cdddeb

(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)

Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-27 18:46:31 +02:00
Romain Naour
2ed3b0a586 gitlab-ci: check all defconfigs on every push
The defconfig check has been introduced by the previous
patch before the building each defconfig but those builds
are done every week or more.

Checking if a defconfig is valid can be done on every
push in the repository since it take few seconds.

This would allow to detect as soon as possible a problem
in a defconfig and eventually avoid breaking the build
while build testing all defconfig.

Introduce a new job template ".defconfig_check" in
gitlab-ci.yml.in and modify the generate-gitlab-ci-yml
to create a job for each defconfig to run the test.

Although, we could have used only one job to do all
tests, using one job per defconfig allow to identify
easily in gitlab which defconfig is falling.

Tested:
https://gitlab.com/kubu93/buildroot/pipelines/138331069
https://gitlab.com/kubu93/buildroot/pipelines/171223758

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 18:13:45 +02:00
Romain Naour
7385ccab18 support/scripts: add check-dotconfig.py
For the same reason as for 50b747f212,
we need to check if the generated configuration file (.config)
contains all symbols present in the defconfig file.

If not there is an issue with the defconfig.

This script will be used in .gitlab-ci.yml.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr:
  - strip defconfig lines when reading them
  - use a generator to read the defconfig lines
  - no need to strip() again when building the missing list
  - testing the list directly, not its len()
  - simply sys.exit(1) in the error condition
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-07-27 18:13:19 +02:00
Thomas Petazzoni
204d03ae43 support/scripts/pkg-stats: fix flake8 warning
This fixes the following flake8 warning:

support/scripts/pkg-stats:1005:9: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:23:13 +02:00
Gregory CLEMENT
7d2779ecbb support/script/pkg-stats: handle exception when version comparison fails
With python 3, when a package has a version number x-y-z instead of
x.y.z, then the version returned by LooseVersion can't be compared
which raises a TypeError exception:

Traceback (most recent call last):
  File "./support/scripts/pkg-stats", line 1062, in <module>
    __main__()
  File "./support/scripts/pkg-stats", line 1051, in __main__
    check_package_cves(args.nvd_path, {p.name: p for p in packages})
  File "./support/scripts/pkg-stats", line 613, in check_package_cves
    if pkg_name in packages and cve.affects(packages[pkg_name]):
  File "./support/scripts/pkg-stats", line 386, in affects
    return pkg_version <= cve_affected_version
  File "/usr/lib64/python3.8/distutils/version.py", line 58, in __le__
    c = self._cmp(other)
  File "/usr/lib64/python3.8/distutils/version.py", line 337, in _cmp
    if self.version < other.version:
TypeError: '<' not supported between instances of 'str' and 'int'

This patch handles this exception by adding a new return value when
the comparison can't be done. The code is adjusted to take of this
change. For now, a return value of CVE_UNKNOWN is handled the same way
as a CVE_DOESNT_AFFECT return value, but this can be improved later
on.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:22:55 +02:00
Thomas Petazzoni
45b174c8ad support/scripts/pkg-stats: remove debug cruft
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 23:30:52 +02:00
Peter Seiderer
185398f619 package/python-colorzero: new package
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 21:45:00 +02:00