Commit Graph

98 Commits

Author SHA1 Message Date
Thomas Petazzoni
40bb37bd70 utils/getdeveloperlib.py: use relative paths for files
Using absolute paths within getdeveloperlib isn't very sensible, it
makes a lot more sense to handle everything as relative paths from the
top-level Buildroot source directory.

parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of
utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to
use relative paths, and prepend them with the base "brpath" when
needed.

This commit allows pkg-stats to report correct developers information
even when executed from an out of tree directory.

Before this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[]

$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": []
}

After this patch:

$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json
$ cat out.json | jq '.packages.ipmitool.developers'
[
  "Floris Bos <bos@je-eigen-domein.nl>",
  "Heiko Thiery <heiko.thiery@gmail.com>"
]
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
  "name": "stm32f469_disco",
  "path": "configs/stm32f469_disco_defconfig",
  "developers": [
    "Christophe Priouzeau <christophe.priouzeau@st.com>"
  ]
}

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:08 +01:00
Thomas Petazzoni
ffb2620405 support/scripts/pkg-stats: fix Python 3.8 deprecation warning
With Python 3.8, the following deprecation warnings are emitted:

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:418: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

/home/thomas/projets/buildroot/./support/scripts/pkg-stats:536: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

The correct way to pass coroutines is to use asyncio.create_task(),
but this is rather new method (Python 3.7), and using it breaks
compatibility with older Python versions. As suggested at
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task,
use the more cryptic, but also more compatible asyncio.ensure_future()
method.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:05 +01:00
Thomas Petazzoni
824032d168 support/scripts/pkg-stats: import cve module only when needed
The cve module needs ijson, which may not be installed. Since cve
matching is only enabled when --nvd-path is passed, it is a bit silly
to error out about ijson being missing if it's not used.

So instead of unconditionally importing the cve module, only do it
conditionally.

However, instead of doing it right at the point where it is used, we
do it at the beginning of the main() function. Indeed, if the cve
module is needed but cannot be imported, we want to error out
immediately rather than doing a whole bunch of things, and failing on
the user later on in the middle of the pkg-stats execution.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-17 15:32:04 +01:00
Arnout Vandecappelle (Essensium/Mind)
24dc403be3 support/scripts/pkg-stats: fix flake8 errors
support/scripts/pkg-stats:81:22: E211 whitespace before '('
support/scripts/pkg-stats:404:1: E305 expected 2 blank lines after class or function definition, found 1
support/scripts/pkg-stats:561:12: E713 test for membership should be 'not in'
support/scripts/pkg-stats:567:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:595:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1051:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:1057:1: E302 expected 2 blank lines, found 1

Also fix:
support/scripts/pkg-stats:1054:5: E722 do not use bare 'except'
found by a more recent flake8 version. The exception may be either
IndexError or AttributeError, so use Exception to catch either.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2021-01-05 23:34:28 +01:00
Thomas Petazzoni
bd665d182c support/scripts/pkg-stats: improve rendering of CVE information
This commit improves pkg-stats to fill in pkg.status['cve'] depending
on the situation for CVEs affecting this package. They are then used
in the HTML rendering.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:39:11 +01:00
Thomas Petazzoni
78d7521f82 support/scripts/pkg-stats: ignore packages with no valid infra and no version for CVE checking
Virtual packages (with in pkg-stats speak have "no valid
infrastructure") and packages that have no version specified cannot be
used for CVE checking. They trigger a bunch of warnings from the CVE
checking code, as it cannot parse their version: they don't have any
version. So instead, we simply skip those packages.

A follow-up commit will improve the reporting to be able to
distinguish those packages from packages that have seen their CVEs
checked and don't have any reported.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:38:35 +01:00
Thomas Petazzoni
e3ef352ef6 support/scripts/{pkg-stats, cve.py}: support CPE ID based matching
This commit modifies cve.py, as well as its users cve-checker and
pkg-stats to support CPE ID based matching, for packages that have CPE
ID information.

One of the non-trivial thing is that we can't simply iterate over all
CVEs, and then iterate over all our packages to see which packages
have CPE ID information that match the CPEs affected by the
CVE. Indeed, this is an O(n^2) operation.

So instead, we do a pre-filtering of packages potentially affected. In
check_package_cves(), we build a cpe_product_pkgs dict that associates
a CPE product name to the packages that have this CPE product
name. The CPE product name is either derived from the CPE information
provided by the package if available, and otherwise we use the package
name, which is what was used prior to this patch.

And then, when we look at CVEs, we only consider the packages that
have a CPE product name matching the CPE products affected by the
CVEs. This is done in check_package_cve_affects().

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:38:20 +01:00
Gregory CLEMENT
92e7089a8c support/script/pkg-stats: show CPE ID in results
This commit improves the pkg-stats script to show the CPE ID of
packages, if available. For now, it doesn't use CPE IDs to match CVEs.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-01-04 21:36:50 +01:00
Thomas Petazzoni
7a607dab33 support/scripts/pkg-stats: support generating stats based on configured packages
pkg-stats was initially a Buildroot maintenance oriented tool: it was
designed to examine all Buildroot packages and provide
statistics/details about them.

However, it turns out that a number of details provided by pkg-stats,
especially CVEs, are relevant also for Buildroot users, who would like
to check regularly if their specific Buildroot configuration is
affected by CVEs or not, and possibly check if all packages have
license information, license files, etc.

The cve-checker script was recently introduced to provide an output
relatively similar to pkg-stats, but focused on CVEs only.

But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses
"make show-info" to only consider packages enabled in the current
configuration.

So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
to generate its output based on the list of configured packages. -c is
mutually exclusive with the -p option (explicit list of packages) and
-n option (a number of packages, picked randomly).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-11 12:03:55 +01:00
Thomas Petazzoni
ae86067a15 support/scripts/pkg-stats: allow to run script outside of the top-level directory
Currently, pkg-stats expects being executed from Buildroot's top-level
source directory. As we are going to extend pkg-stats to cover only
the packages available in the current configuration, it makes sense to
be able to run it from the output directory, which can be anywhere
compared to Buildroot's top-level directory.

This commit adjusts pkg-stats to this, by inferring all Buildroot
paths based on the location of the pkg-stats script itself.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-11 12:01:21 +01:00
Yann E. MORIN
843334bf4f support/scripts: fix flake8 errors in cve-checker and pkg-stats
Note that one is silenced, rather than fixed: we indeed need to import
after we add the local directory to the modules search path.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-09-03 20:48:12 +02:00
Bernd Kuhls
6b2355fef0 package/nvidia-tegra23: remove package
Current X.org X server is incompatible with this driver.

We no longer support unmaintainted versions of X.org X server.

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2020-09-01 21:57:21 +02:00
Gregory CLEMENT
2a2f69d672 support/scripts: make CVE class independent of the Packaage class
The affects method of the CVE uses the Package class defined in
pkg-stats. The purpose of migrating the CVE class outside of pkg-stats
was to be able to reuse it from other scripts. So let's remove the
Package dependency and only use the needed information.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 11:00:40 +02:00
Gregory CLEMENT
f8ee65570a support/scripts/pkg-stats: move CVE related code to a separate module
In order to be able to use the CVE checking logic outside of
pkg-stats, move the CVE class in a module that can be used by other
scripts.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
008ca2c583 support/scripts/pkg-stats: consider "-" as a wildcard when doing CVE version matching
Some CVE entries in the NVD database have version_value set to "-",
which seems to indicate that it applies to all versions of the
software project, or that they don't really know which versions are
affected, and which are not.

So, for the benefit of doubt, it seems more appropriate to consider
such CVEs as affecting our packages.

This makes the total number of CVEs affecting our next branch jump
from 141 CVEs to 658 CVEs, but that number will go back down once we
switch to the JSON 1.1 schema. Indeed, in the JSON 1.0 schema, there
are often cases where a version_value is set to "=" *and* specific
versions are set to.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
e631c314b1 support/scripts/pkg-stats: drop erroneous "break" in CVE.affects()
Commit 7d2779ecbb
("support/script/pkg-stats: handle exception when version comparison
fails") erroneousy introduced a "break" within a try/expect block.

This break has the unfortunate consequence that every CVE that was
using the <= operator was skipped, and according to the current
CVE statistics, made us miss 74 CVEs out of 141 CVEs.

Here is for reference the complete list of CVEs we missed:

 - gnupg
   CVE-2006-3082
   CVE-2019-13050

 - jhead
   CVE-2020-6624
   CVE-2020-6625

 - patch
   CVE-2018-6952
   CVE-2019-20633

 - json-c
   CVE-2020-12762

 - git
   CVE-2018-1000110
   CVE-2018-1000182
   CVE-2019-1003010
   CVE-2020-2136

 - iperf2
   CVE-2016-4303

 - libtorrent
   CVE-2009-1760
   CVE-2016-5301

 - lua
   CVE-2020-15888
   CVE-2020-15889
   CVE-2020-15945
   CVE-2020-24342

 - openvpn
   CVE-2020-7224

 - smack
   CVE-2016-10027

 - bashtop
   CVE-2019-18276

 - links
   CVE-2008-3319

 - argus
   CVE-2011-3332

 - libraw
   CVE-2020-15503

 - netcat
   CVE-2008-5727
   CVE-2008-5728
   CVE-2008-5729
   CVE-2008-5730
   CVE-2008-5742
   CVE-2015-2214

 - subversion
   CVE-2017-1000085
   CVE-2018-1000111
   CVE-2020-2111

 - python
   CVE-2013-1753
   CVE-2015-5652
   CVE-2017-17522
   CVE-2017-18207
   CVE-2019-20907
   CVE-2019-9674

 - cereal
   CVE-2020-11104
   CVE-2020-11105

 - opencv
   CVE-2017-1000450
   CVE-2017-12597
   CVE-2017-12598
   CVE-2017-12599
   CVE-2017-12600
   CVE-2017-12601
   CVE-2017-12602
   CVE-2017-12603
   CVE-2017-12604
   CVE-2017-12605
   CVE-2017-12606
   CVE-2017-12862
   CVE-2017-12863
   CVE-2017-12864
   CVE-2019-15939

 - docker
   CVE-2015-1843
   CVE-2015-3627
   CVE-2015-3630
   CVE-2015-3631
   CVE-2016-3697
   CVE-2017-14992
   CVE-2019-16884

 - trousers
   CVE-2020-24330
   CVE-2020-24331
   CVE-2020-24332

 - libcroco
   CVE-2020-12825

 - libpupnp
   CVE-2020-13848

 - openjpeg
   CVE-2020-15389

 - flex
   CVE-2015-1773

 - libesmtp
   CVE-2019-19977

 - ed
   CVE-2015-2987

 - libmad
   CVE-2018-7263

 - grub
   CVE-2020-15705

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-28 09:14:38 +02:00
Thomas Petazzoni
4f0868fa64 support/scripts/pkg-stats: show progress of upstream URL and latest version
This commit slightly improves the output of pkg-stats by showing the
progress of the upstream URL checks and latest version retrieval, on a
package basis:

Checking URL status
[0001/0062] curlpp
[0002/0062] cmocka
[0003/0062] snappy
[0004/0062] nload
[...]
[0060/0062] librtas
[0061/0062] libsilk
[0062/0062] jhead
Getting latest versions ...
[0001/0064] libglob
[0002/0064] perl-http-daemon
[0003/0064] shadowsocks-libev
[...]
[0061/0064] lua-flu
[0062/0064] python-aiohttp-security
[0063/0064] ljlinenoise
[0064/0064] matchbox-lib

Note that the above sample was run on 64 packages. Only 62 packages
appear for the URL status check, because packages that do not have any
URL in their Config.in file, or don't have any Config.in file at all,
are not checked and therefore not accounted.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:52 +02:00
Thomas Petazzoni
c8ba6aa133 support/scripts/pkg-stats: use aiohttp for upstream URL checking
This commit reworks the code that checks if the upstream URL of each
package (specified by its Config.in file) using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:43 +02:00
Thomas Petazzoni
d28b2bc481 support/scripts/pkg-stats: use aiohttp for latest version retrieval
This commit reworks the code that retrieves the latest upstream
version of each package from release-monitoring.org using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Since we're now using some async functionality, the script is Python
3.x only, so the shebang is changed to make this clear.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-11 22:30:25 +02:00
Thomas Petazzoni
204d03ae43 support/scripts/pkg-stats: fix flake8 warning
This fixes the following flake8 warning:

support/scripts/pkg-stats:1005:9: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:23:13 +02:00
Gregory CLEMENT
7d2779ecbb support/script/pkg-stats: handle exception when version comparison fails
With python 3, when a package has a version number x-y-z instead of
x.y.z, then the version returned by LooseVersion can't be compared
which raises a TypeError exception:

Traceback (most recent call last):
  File "./support/scripts/pkg-stats", line 1062, in <module>
    __main__()
  File "./support/scripts/pkg-stats", line 1051, in __main__
    check_package_cves(args.nvd_path, {p.name: p for p in packages})
  File "./support/scripts/pkg-stats", line 613, in check_package_cves
    if pkg_name in packages and cve.affects(packages[pkg_name]):
  File "./support/scripts/pkg-stats", line 386, in affects
    return pkg_version <= cve_affected_version
  File "/usr/lib64/python3.8/distutils/version.py", line 58, in __le__
    c = self._cmp(other)
  File "/usr/lib64/python3.8/distutils/version.py", line 337, in _cmp
    if self.version < other.version:
TypeError: '<' not supported between instances of 'str' and 'int'

This patch handles this exception by adding a new return value when
the comparison can't be done. The code is adjusted to take of this
change. For now, a return value of CVE_UNKNOWN is handled the same way
as a CVE_DOESNT_AFFECT return value, but this can be improved later
on.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-07-12 21:22:55 +02:00
Thomas Petazzoni
45b174c8ad support/scripts/pkg-stats: remove debug cruft
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 23:30:52 +02:00
Peter Seiderer
185398f619 package/python-colorzero: new package
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-06-18 21:45:00 +02:00
Heiko Thiery
f41056ec4b support/scripts/pkg-stats: add tilde '~' expansion for pathes
When the 'nvd-path', 'json' and 'html' are used like this:

  --html ~/foo

then the tilde expansion is properly done by the shell. However, when
they are used like this:

  --html=~/foo

The shell doesn't do the tilde expansion, and pkg-stats doesn't do
it. This commit modifies pkg-stats to ensure that tilde expansion is
done when parsing the 'nvd-path', 'json' and 'html' arguments.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: improve commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-12 14:49:45 +02:00
Thomas Petazzoni
3b5bc480a5 support/scripts/pkg-stats: fix flake8 E722 warning
flake8 complains with:

  support/scripts/pkg-stats:339:13: E722 do not use bare 'except'

Due to the construct:

  try:
     something
  except:
     print("some message")
     raise

Which is in fact OK because the exception is re-raised. This issue is
discussed at https://github.com/PyCQA/pycodestyle/issues/703, and the
general agreement is that these "bare except" are OK, and should be
ignored from flake8 using a noqa statement.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
f7f33771b3 support/scripts/pkg-stats: fix flake8 E501 warning
Fixes:

support/scripts/pkg-stats:281:133: E501 line too long (139 > 132 characters)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
198d76efb3 support/scripts/pkg-stats: fix flake8 E117 warning
Fixes:

  support/scripts/pkg-stats:146:17: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
e03bdef0ec support/scripts/pkg-stats: fix flake8 E302 warning
Fixes:

  support/scripts/pkg-stats:57:1: E302 expected 2 blank lines, found 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:57 +01:00
Thomas Petazzoni
769f98c18c support/scripts/pkg-stats: fix flake8 E402 warning
flake8 complains with:

pkg-stats:38:1: E402 module level import not at top of file

This is due to sys.path.append() being before the import from
getdeveloperlib, but we really need this sys.path.append() to be
before, so let's ignore this flake8 warning.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-24 15:27:53 +01:00
Peter Korsgaard
fd99eb5016 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-09 15:17:09 +01:00
Heiko Thiery
759521dae6 support/scripts/pkg-stats: add list of status checks to the json output
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:38:26 +01:00
Heiko Thiery
fb879c1954 support/scripts/pkg-stats: set status to 'na' for virtual packages
If there is no infra set or infra is virtual the status is set to 'na'.

This is done for the follwing checks:
 - license
 - license-files
 - hash
 - hash-license
 - patches
 - version

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:37:23 +01:00
Heiko Thiery
8d77ecbad0 support/scripts/pkg-stats: add defconfig support
Scan configs directory and create Defconfig objects.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:37:02 +01:00
Heiko Thiery
d31fadfbf5 support/scripts/pkg-stats: store pkg dir path
This value can be used for later processing.

In the buildroot-stats application this is used to create links pointing
to the git repo of buildroot.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:54 +01:00
Heiko Thiery
0e267518cb support/scripts/pkg-stats: add package count to stats
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:45 +01:00
Heiko Thiery
f422fa991f support/scripts/pkg-stats: add package status
Unify the status check information. The status is stored in a tuple. The
first entry is the status that can be 'ok', 'warning' or 'error'. The
second entry is a verbose message.

The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check

With that status information the following variables are replaced:
has_license, has_license_files, has_hash, url_status

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:38 +01:00
Heiko Thiery
5b7278e5f1 support/scripts/pkg-stats: store licences of package
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:17 +01:00
Heiko Thiery
c1fc827934 support/scripts/pkg-stats: set developers info
Use the function 'parse_developers' function from getdeveloperlib that
collect the information about the developers and the files they
maintain. Then set the maintainer(s) to each package.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 21:36:08 +01:00
Heiko Thiery
b1916b0a8d support/scripts/pkg-stats: store patch files for the package
Remove the patch_count attribute and use a class property instead.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 15:59:52 +01:00
Heiko Thiery
c46e707182 support/scripts/pkg-stats: store latest version in a dict
This patch changes the type of the latest_version variable to a dict.
This is for better readability/usability of the data. With this the json
output is more descriptive in later processing of the json output.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-03-07 15:59:14 +01:00
Titouan Christophe
28adf09b89 support/scripts/pkg-stats: clear multiprocessing pools after use
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:59:08 +01:00
Titouan Christophe
fb05ab2242 support/scripts/pkg-stats: decode subprocess output for python3
In Python 3, the functions from the subprocess module return bytes
(and no longer strings as in Python 2), which must be decoded for
further text operations.

Now, pkg-stats can be run in Python 3.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:59:04 +01:00
Thomas Petazzoni
1097c0427d support/scripts/pkg-stats: properly ignore CVEs in <pkg>_IGNORE_CVES
It seems like throughout the series that the CVE pkg-stats support
went through, the support for ignoring CVEs in the per-package
<pkg>_IGNORE_CVES variable was forgotten.

Let's re-introduce this, which is now very simple thanks to the CVE
class, its .identifier() propertly and the .is_cve_ignored() method of
the Package class

Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-07 15:58:41 +01:00
Titouan Christophe
54645c0b39 support/scripts/pkg-stats: clear multiprocessing pools after use
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-02 23:35:39 +01:00
Titouan Christophe
304b141a97 support/scripts/pkg-stats: decode subprocess output for python3
In Python 3, the functions from the subprocess module return bytes
(and no longer strings as in Python 2), which must be decoded for
further text operations.

Now, pkg-stats can be run in Python 3.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-02 23:35:26 +01:00
Titouan Christophe
a35f51cee1 support/scripts/pkg-stats: iterate over CVEs in streaming
The NVD files that are used to build the list of CVEs affecting
Buildroot packages are quite large (a few hundreds MB of json),
and cause the pkg-stats scripts to have a huge memory footprint
(a few GB with Python 2.7).

However, because we only need to iterate on CVE items one by one,
we can process them in streaming (ie decoding one CVE at a time
from the JSON representation). Because the json module from the
python standard library does not support such a mode of operation,
we switch to the third-party package ijson, which is compatible
with both Python 2 and Python3.

To run the script with these modifications, one should install
the ijson python package. This can be done with pip:
`pip install ijson`. On Debian based distributions, this can
also be done with the apt package manager:
`apt install python-ijson`.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-24 22:22:58 +01:00
Titouan Christophe
712f81c41c support/scripts/pkg-stats: iterate over CVEs in streaming
The NVD files that are used to build the list of CVEs affecting
Buildroot packages are quite large (a few hundreds MB of json),
and cause the pkg-stats scripts to have a huge memory footprint
(a few GB with Python 2.7).

However, because we only need to iterate on CVE items one by one,
we can process them in streaming (ie decoding one CVE at a time
from the JSON representation). Because the json module from the
python standard library does not support such a mode of operation,
we switch to the third-party package ijson, which is compatible
with both Python 2 and Python3.

To run the script with these modifications, one should install
the ijson python package. This can be done with pip:
`pip install ijson`. On Debian based distributions, this can
also be done with the apt package manager:
`apt install python-ijson`.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-20 21:31:05 +01:00
Thomas Petazzoni
60f2de1f12 support/scripts/pkg-stats: properly ignore CVEs in <pkg>_IGNORE_CVES
It seems like throughout the series that the CVE pkg-stats support
went through, the support for ignoring CVEs in the per-package
<pkg>_IGNORE_CVES variable was forgotten.

Let's re-introduce this, which is now very simple thanks to the CVE
class, its .identifier() propertly and the .is_cve_ignored() method of
the Package class

Cc: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-19 08:22:09 +01:00
Thomas Petazzoni
4a157be9ef support/scripts/pkg-stats: add support for CVE reporting
This commit extends the pkg-stats script to grab information about the
CVEs affecting the Buildroot packages.

To do so, it downloads the NVD database from
https://nvd.nist.gov/vuln/data-feeds in JSON format, and processes the
JSON file to determine which of our packages is affected by which
CVE. The information is then displayed in both the HTML output and the
JSON output of pkg-stats.

To use this feature, you have to pass the new --nvd-path option,
pointing to a writable directory where pkg-stats will store the NVD
database. If the local database is less than 24 hours old, it will not
re-download it. If it is more than 24 hours old, it will re-download
only the files that have really been updated by upstream NVD.

Packages can use the newly introduced <pkg>_IGNORE_CVES variable to
tell pkg-stats that some CVEs should be ignored: it can be because a
patch we have is fixing the CVE, or because the CVE doesn't apply in
our case.

>From an implementation point of view:

 - A new class CVE implement most of the required functionalities:
   - Downloading the yearly NVD files
   - Reading and extracting relevant data from these files
   - Matching Packages against a CVE

 - The statistics are extended with the total number of CVEs, and the
   total number of packages that have at least one CVE pending.

 - The HTML output is extended with these new details. There are no
   changes to the code generating the JSON output because the existing
   code is smart enough to automatically expose the new information.

This development is a collective effort with Titouan Christophe
<titouan.christophe@railnova.eu> and Thomas De Schampheleire
<thomas.de_schampheleire@nokia.com>.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-15 16:49:07 +01:00
Thomas Petazzoni
ffcd34af07 support/scripts/pkg-stats: simplify Git commit id retrieval
As suggested by Baruch Siach, using "git rev-parse HEAD" is a lot
simpler than playing around with "git log" to just retrieve the commit
id corresponding to the current HEAD.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-15 16:10:22 +02:00