When the CVE lookup was added in commit
4a157be9ef, the starting year of the JSON
files was set to 2002. However, there are also CVEs from 1999, 2000 and
2001. It is not clear why these were skipped back then.
Set the start year to 1999 to capture these old CVEs too.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
NVD will deprecate the v1.1 API which allows us to download the full
database as individual JSON files. Instead, there's a horribly crappy
API that is extremely slow and subject to race conditions.
Fortunately, there is a project, Fraunhofer FKIE - Cyber Analysis and
Defense [1], that goes through the effort of adapting to this new API
and regenerating the convenient JSON files. The JSON files and meta
files are re-generated daily.
Instead of implementing the NVD v2 API, we decided to just use the JSON
files generatd by fkie-cad. That saves us the effort of solving the race
conditions, devising a cache mechanism that works, handling the frequent
gateway timeouts on the NVD servers, dealing with the rate limiting, and
keeping up with changes in the API.
Switch to this repository on github as NVD_BASE_URL. The file name is
also slightly different (CVE-20XX.json instead of nvdcve-1.1-20XX.json).
The fkie-cad repository compresses with xz instead of gz. Therefore:
- rename the filename variables to _xz instead of _gz;
- use xz as a subprocess because there is no xz decompressor in Python
stdlib.
[1] https://www.fkie.fraunhofer.de/en/departments/cad.html
Cc: Daniel Lang <dalang@gmx.at>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Python 2 is EOL sice 2020 [1], it's still available on distros, but may not
be installed by default (as being replaced by python3).
Thus remove compatibility imports:
from __future__ import print_function
from __future__ import absolute_import
Tested with python3 -m py_compile.
[1] https://www.python.org/doc/sunset-python-2/
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
ijson < 2.5 (as available in Debian 10) use the slow python backend by
default instead of the most efficient one available like modern ijson
versions, significantly slowing down cve checking. E.G.:
time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total
To
93,53s user 2,00s system 98% cpu 1:36,65 total
E.G. almost 2x as fast.
As a workaround, detect when the python backend is used and try to use a
more efficient one instead. Use the yajl2_cffi backend as recommended by
upstream, as it is most likely to work, and print a warning (and continue)
if we fail to load it.
The detection is slightly complicated by the fact that ijson.backends used
to be a reference to a backend module, but is nowadays a string (without the
ijson.backends prefix).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This commit modifies cve.py, as well as its users cve-checker and
pkg-stats to support CPE ID based matching, for packages that have CPE
ID information.
One of the non-trivial thing is that we can't simply iterate over all
CVEs, and then iterate over all our packages to see which packages
have CPE ID information that match the CPEs affected by the
CVE. Indeed, this is an O(n^2) operation.
So instead, we do a pre-filtering of packages potentially affected. In
check_package_cves(), we build a cpe_product_pkgs dict that associates
a CPE product name to the packages that have this CPE product
name. The CPE product name is either derived from the CPE information
provided by the package if available, and otherwise we use the package
name, which is what was used prior to this patch.
And then, when we look at CVEs, we only consider the packages that
have a CPE product name matching the CPE products affected by the
CVEs. This is done in check_package_cve_affects().
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the version encoded in a CPE is '-', we assume all
versions are affected, but when it's '*' with no further range
information, we assume no version is affected.
This doesn't make sense, so instead, we handle '*' and '-' in the same
way. If there's no version information available in the CVE CPE ID, we
assume all versions are affected.
This increases quite a bit the number of CVEs and package affected:
- "total-cves": 302,
- "pkg-cves": 100,
+ "total-cves": 597,
+ "pkg-cves": 135,
For example, CVE-2007-4476 has a CPE ID of:
cpe:2.3🅰️gnu:tar:*:*:*:*:*:*:*:*
So it should be taken into account. In this specific case, it is
combined with an AND with CPE ID
cpe:2.3⭕suse:suse_linux:10:*:enterprise_server:*:*:*:*:* but since
we don't support this kind of matching, we'd better be on the safe
side, and report this CVE as affecting tar, do an analysis of the CVE
impact, and document it in TAR_IGNORE_CVES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The affects method of the CVE uses the Package class defined in
pkg-stats. The purpose of migrating the CVE class outside of pkg-stats
was to be able to reuse it from other scripts. So let's remove the
Package dependency and only use the needed information.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In 2019, the JSON vulnerability feeds switched their schema from
version 1.0 to 1.1.
The main difference is the removal of the "affects" element that we
were using to check if a package was affected by a CVE.
This information is now available in the "configuration" element which
contains the cpeid as well as properties about the versions
affected. Instead of having a list of the versions affected, with
these properties, it is possible to have a range of versions.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In order to be able to use the CVE checking logic outside of
pkg-stats, move the CVE class in a module that can be used by other
scripts.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
