fee7efafd0
Currently, we grab the per-year CVE feeds, in two passes: first, we grab the meta files, and check whether something has changed since last we downloaded it; second, we download the feed proper, unless the meta file has not changed, in which case we use the locally cached feed. However, it has appeared that the FKIE releases no longer provide the meta files, which means that (once again), our daily reports are broken. The obvious fix would be to drop the use of the meta file, and always and unconditionally download the feeds. That's relatively trivial to do, but the feeds are relatively big (even as xz-xompressed). However, the CVE database from FKIE is available as a git tree. Git is pretty good at only sending delta when updating a local copy. In addition, the git tree, contains each CVE as an individual file, so it is relatively easier to scan and parse. Switch to using a local git clone. Slightly surprisingly (but not so much either), parsing the CVE files is much faster when using the git working copy, than it is when parsing the per-year feeds: indeed, the per-year feeds are xz-compressed, and even if python is slow-ish to scan a directory and opening files therein, it is still much faster than to decompress xz files. The timing delta [0] is ~100s before and ~10s now, about a ten time improvement, over the whole package set. The drawback, however, is that the git tree is much bigger on-disk, from ~55MiB for the per-year compressed feeds, to 2.1GiB for the git tree (~366MiB) and a working copy (~1.8GiB)... Given very few people are going to use that, that's considered acceptable... Eventually, with a bit of hacking [1], the two pkg-stats, before and after this change, yield the same data (except for the date and commit hash). [0] hacking support/scripts/pkg-stats to display the time before/after the CVE scan, and hacking support/scripts/cve.py to do no download so that only the CVE scan happens (and also because the meta files are no longer available). [1] sorting the CVE lists in json, sorting the json keys, and using the commit from the FKIE git tree that was used for the current per-year feeds. Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Arnout Vandecappelle <arnout@mind.be> |
||
|---|---|---|
| arch | ||
| board | ||
| boot | ||
| configs | ||
| docs | ||
| fs | ||
| linux | ||
| package | ||
| support | ||
| system | ||
| toolchain | ||
| utils | ||
| .checkpackageignore | ||
| .clang-format | ||
| .defconfig | ||
| .editorconfig | ||
| .flake8 | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .shellcheckrc | ||
| CHANGES | ||
| Config.in | ||
| Config.in.legacy | ||
| COPYING | ||
| DEVELOPERS | ||
| Makefile | ||
| Makefile.legacy | ||
| README | ||
Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation. The documentation can be found in docs/manual. You can generate a text document with 'make manual-text' and read output/docs/manual/manual.text. Online documentation can be found at http://buildroot.org/docs.html To build and use the buildroot stuff, do the following: 1) run 'make menuconfig' 2) select the target architecture and the packages you wish to compile 3) run 'make' 4) wait while it compiles 5) find the kernel, bootloader, root filesystem, etc. in output/images You do not need to be root to build or run buildroot. Have fun! Buildroot comes with a basic configuration for a number of boards. Run 'make list-defconfigs' to view the list of provided configurations. Please feed suggestions, bug reports, insults, and bribes back to the buildroot mailing list: buildroot@buildroot.org You can also find us on #buildroot on OFTC IRC. If you would like to contribute patches, please read https://buildroot.org/manual.html#submitting-patches