Any .pyc files generated by the pycompile script during target
finalization are currently counted in the "Unknown" package,
because packages-file-list.txt only contains the source .py file.
If a .py file is added to filesdict, add the corresponding .pyc
file as well.
Signed-off-by: Michael Klein <m.klein@mvz-labor-lb.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes flake8 errors:
support/scripts/pkg-stats:1013:133: E501 line too long (164 > 132 characters)
support/scripts/pkg-stats:1018:36: F541 f-string is missing placeholders
support/scripts/pkg-stats:1110:199: E261 at least two spaces before inline comment
For the first and the last one, we chose to split the long lines rather
than adding noqa: 501. Indeed, the long lines make it very unreadable,
and there are relatively natural places where the line can be broken.
Also split a line just below the second one in a similar way.
The f-string on 1018 doesn't need to be an f-string.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
This is done either by switching to single quoted f-strings, triple
double quoted f-strings when needed, or simply single-quoted strings.
The renderer HTML is exactly identical before/after this commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
Within single-quoted f-strings, and within triple double quoted
strings, escaping all the double quotes is completely useless and
makes the code more difficult to read. Get rid of all this useless
escaping.
The renderer HTML is exactly identical before/after this commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
The python installer package isn't able to overwrite files of packges
that already exist, this causes problems when doing a rebuild or
update without a full clean.
To fix this we can use functionality from importlib to identify and
remove any conflicting python package files before installation.
We also need to use internals from python-installer, as we want to use
the same logic as pyinstaller uses internally for getting the scheme so
that we ensure we clean the correct package scheme (we want it to be the
same as the one we're installing)
Fixes:
Traceback (most recent call last):
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
main()
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
install(
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
record = destination.write_file(
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
return self.write_to_fs(scheme, path_, stream, is_executable)
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
[yann.morin.1998@free.fr:
- extend commit log about the use of the installer internals (the
symbols prefixed with '_')
- check path.files against explicitly None
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The python installer cli isn't able to overwrite files of packages
that already exist, this causes problems when doing a rebuild or
update without a full clean.
Since we need to add functionality to our pyinstaller.py script to fix
this issue we must also use pyinstaller.py for host python packages.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This adds an optional linebreak at the vendor attribute in the CPE ID.
It should be noted this is purely for formatting/layout purposes
and does not actually insert any additional characters
(newline or otherwise) into the rendered text.
This means that even though the text renders across two lines,
copy-pasting will still yield one line of text.
example: https://sen-h.github.io/pkg-stats/c245575.html
see also: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/wbr
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This attempts to re-implement the "sortable_hint" feature without
relying on words. The column headers and CVE expand/contract buttons
change color and cursor style on hover.
If Javascript is enabled:
Just like [PATCH 1/3] more rules are applied to the
generated stylesheet before content is loaded.
If Javascript is disabled:
The headers stay pearly white. :-)
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Sometimes a package can have a lot of CVEs.
Rather than have the CVE cell make a really tall row
(that means you have to scroll a bunch) this collapses the CVE
cell to a fixed size scrollable element with a
sticky button that lets you expand and collapse it.
If Javascript is enabled:
A stylesheet is generated and appended before content rendering,
amending the cells style to have a fixed height and overflow.
Also, the expand/contract button is unhidden.
This means the CVE cells are rendered in a collapsed state
instead of being rendered in an expanded state and then
slamming shut.
This avoids a "flash" and *helps* (vertically at least) manage CLS
(cumulative layout shift).
see: https://web.dev/cls/
If Javascript is disabled:
The cells stay fully open and the expand/contract button stays hidden.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit c4e6d5c8be (core: implement per-package SDK and target)
introduced leading TABs in that otherwise-space-indented script.
Convert all to spaces, for homogeneity
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Back many years ago, we developed an Eclipse plugin that simplified
the usage of Buildroot toolchains. Enabling the BR2_ECLIPSE_REGISTER=y
was registering the Buildroot toolchain into a special file in your
HOME folder that the Eclipse plugin would recognize to allow to
directly use the Buildroot cross-compiler.
This Eclipse plugin has not been maintained for years. The last commit
in the repository dates back from September 2017. Since then Eclipse
has moved on, and the plugin is no longer compatible with current
versions of Eclipse.
Also, Eclipse is probably no longer that widely used in the embedded
Linux space, as other more modern IDEs have become more popular.
All in all, it's time to say good bye to this Eclipse integration.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
sortGrid() has been rewritten to dynamically generate stylesheets with
explicit grid-row properties to re-order the rows, instead of removing
and reinserting the cells.
Performance *should* now be comperable to sorttable.js.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Like all good problems, disparate pieces work together to create
a "synergistically" hairy mess.
The sortGrid() overhaul highlighted a flaw in pkg-stats allowing
for duplicate package class names across rows.
As an example,
boot/barebox/barebox.mk and boot/barebox/barebox/barebox.mk
both get the classname ._barebox and so sortGrid() sticks them on
the same line giving a table with a vestigal row sticking out
of the right side like some kind of appendage.
Also I neglected to add a "_" to the current version column's cells
pkgname class so instead of "._pkgname" we had ".pkgname" and so
the cells were not collected properly as part of the row.
These issues explain the formatting weirdness.
package classnames are now ".path_to_package_makefile" without suffix
(.mk) (so ._boot_barebox_barebox and ._boot_barebox_barebox_barebox
instead of ._barebox) in order to guarantee uniqueness.
and what was *accidentally*
class="centered current_version data .barebox" is now
class="centered current_version data ._boot_barebox_barebox"
just like *all* the other cells in the row. :p
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 'rtl8723ds' name, when queried from release-monitoring.org at
https://release-monitoring.org/api/projects/?pattern=rtl8723ds returns
one project, with one "stable_versions" array, which is empty. This
was not expected by the pkg-stats code, causing an exception:
Task exception was never retrieved
future: <Task finished name='Task-764' coro=<check_package_latest_version_get() done, defined at /home/thomas/projets/buildroot/./support/scripts/pkg-stats:558> exception=IndexError('list index out of range')>
Traceback (most recent call last):
File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 566, in check_package_latest_version_get
if await check_package_get_latest_version_by_guess(session, pkg):
File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 544, in check_package_get_latest_version_by_guess
projects[0]['stable_versions'][0],
IndexError: list index out of range
This commit therefore improves the checks done on the results received
from release-monitoring.org to avoid this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Pretty straightforward. Adds my name to copyright notice.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
CSS classes are generated for each package name for sorting purposes,
However some package names start with a number and this is not allowed.
(see https://www.w3.org/TR/CSS2/syndata.html#value-def-identifier)
Fix is to prepend a character to every class name such as "_".
so every ".package" is now "._package".
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This fixes the .version-needs-update class being overridden by .correct class.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Somewhere between binutils 2.35 and 2.37, some functionality was
added in readelf to parse more DWARF information. Unfortunately, as
reported in binutils bug
28981 ("https://sourceware.org/bugzilla/show_bug.cgi?id=28981"), this
feature causes a number of fairly scary warnings to be displayed when
running readelf on binaries built with Clang, such as the pre-built
rustc and rustdoc binaries part of the host-rust-bin package. It
looks like this:
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 2f in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 10b in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 10c in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Further warnings about bogus end-of-sibling markers suppressed
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: DIE at offset 0x1da refers to abbreviation number 5827 which does not exist
These warnings are caused by the readelf calls done by the
support/scripts/check-host-rpath script. The annoying thing is that
once host-rust-bin has been installed in $(HOST_DIR), this warning
appears after the installation of every single host package, because
support/scripts/check-host-rpath rescans all binaries every time.
To avoid showing those scary warnings, this commit sends the error
output of readelf to /dev/null.
Of course, it would be nicer to only filter out those warnings, but
filtering the error output without merging the error output into the
standard output is tricky, so let's keep things simple. If there is
really an error, readelf will abort.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
As for the Buildroot testsuite, multiply every emulator timeout by 10
to avoid sporadic failures in elastic runners.
qemu_arm_vexpress_tz_defconfig tested locally with sucess.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, glibc depends on !BR2_STATIC_LIBS in all the toolchain
variants.
However, for some architectures, glibc is the only supported libc. In
commit 3b3105328e ("Config.in: only
allow BR2_STATIC_LIBS on supported libc/arch"), we implemented a fix
to avoid configurations were BR2_STATIC_LIBS=y with an architecture
already supported by glibc, because these configurations are
impossible. This commit 3b3105328e
prevents from selecting BR2_STATIC_LIBS=y when the C library used for
the internal toolchain backend is glibc.
However, it introduces a discrepency between how this topic is handled
for internal and external toolchains:
- For internal toolchains, we prevent BR2_STATIC_LIBS=y if glibc is
chosen.
- For external toolchains, we allow BR2_STATIC_LIBS=y in all cases,
and it's each glibc toolchain that has !BR2_STATIC_LIBS
This commit addresses this discrepency by preventing BR2_STATIC_LIBS=y
if glibc is chosen in all cases.
Thanks to this, we can remove the !BR2_STATIC_LIBS dependency on both
the glibc package, and all glibc external toolchains.
Fixes: https://bugs.busybox.net/show_bug.cgi?id=14256
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: update to master, fix the gen-bootlin-toolchains script, add
a comment in the static/shared choice to indicate that static is
supported only with uclibc or musl]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we use the statistics output to generate a CVE/CPE customer
report showing whether a product is affected by CVEs, we are primarily
interested in whether they are relevant to the target
system. Currently we cannot see if the package is configured for the
build (infra==host) and/or the target system (infra==target).
Therefore this commit extends the pkg-stats script to leverage the
information available in "make show-info" output to tweak the list of
package infrastructures for each package. Thanks to this commit, the
script now has a more consistent behavior:
* When pkg-stats is run without -c, i.e without a defined Buildroot
configuration, it continues to operate as it did, i.e it lists all
package infrastructures supported by the package (such as autotools
host+target, or kconfig target, etc.)
* When pkg-stats is run with -c, i.e with a defined Buildroot
configuration which defines the list of packages that should be
considered, then for each package it now lists only the package
infrastructures used by the package in that current
configuration. For example if you have a package with a host and
target variant, but only the host variant is used in your
configuration, now the pkg-stats output will only say that the host
variant of this package is used;
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: pretty much rework the entire implementation and how the
result is presented.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit f1bcb2a45c introduced a number of
flake8 errors. Fix these by:
- adding noqa to the multi-line string containing tabs;
- replacing other tabs with spaces;
- removing space after opening parenthesis;
- splitting the long lines.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This migrates pkg-stats.html from html tables to CSS grid, allowing
the use of newer, simpler javascript that is short enough to be
inlined, instead of relying on externally hosted javascript.
Javascript sorting function was rewritten from scratch in ~55 lines,
short enough to be inlined directly in the html.
Tables were redone in CSS grid, but with care taken to mimic existing
"look and feel" of prevous implementation, albeit with slightly
better responsive behavior and default styling characteristics.
Column labels are now "sticky" and stay stuck to the top of the
viewport as you scroll down the page.
Also, css was rewritten in fewer lines and table elements were changed
to divs (for grid support).
Other small misc fixes include quoted hrefs and document language
declarations to make the w3c html validator happy.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Previously, we used support/scripts/pycompile.py to generate the pyc
files for the python libraries.
While the script worked, it did not follow the PEP 3147 layout
requirements for py+pyc deployments.
Now, use the package's own compileall.py script. This will follow
PEP 3147 guidelines. It also supports "legacy" pyc only deployments as
described here:
https://peps.python.org/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports
With this change, we no longer need to hack support for side-by-side pyc
files because files will be deployed as appropriate.
This also has the added benefit of not requiring python3 on the host to
build host-python3.
Fixes: #14911
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
- build-tested in a python-less environment
- build+run-tested with the runtime-test infra
]
Tested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Buildroot documentation section 9.2.1.6 "Additional kernel extensions"
indicates support for kernel extensions defined in external buildroot
trees but unfortunately, there didn't seem to be any support in
br2-external script.
This patch copies 'init' code support to include external kernel
extensions defined in 'linux' dir at the br2-external root directory as
explained in documentation.
Signed-off-by: Nicolas POIROT <ni.poirot@laposte.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This Bootlin toolchain has been available for a while, but was not
supported until now.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
These toolchains have been marked obsolete by toolchains.bootlin.com
as they are replaced by the mips64-n32 toolchains. See commit:
121e78806b
Will allow to fix:
http://autobuild.buildroot.net/results/d13b35ba5a0f68f72e6592bdd9218b625a3c6554/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The "arches" dict lists the architectures for which Bootlin toolchains
are available, along with the corresponding Buildroot conditions for
those architecture variants.
However, there is nothing that checks that such architectures really
have Bootlin toolchains available. Even if no toolchain is available,
the architecture is considered as support, and will be listed in the
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS option, making menuconfig
believe that some Bootlin toolchain is available for the selected
architecture variant.
This is currently the case with the "mips64" architecture (as
identified by the Bootlin toolchains project). Such toolchains have
been made obsolete in toolchains.bootlin.com and replaced by the
mips64-n32 toolchains. But "mips64" is still listed in "arches",
causing this architecture to be considered as having Bootlin
toolchains available.
To avoid this to happen in the future, this commit adds a check that
verifies that an architecture listed in "arches" really has at least
one toolchain available.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The microblazeel, microblazebe and openrisc toolchains need to select
this option as they don't have any gdb/gdbserver.
Helps in fixing:
http://autobuild.buildroot.net/results/6315ef7b66ee4ae8f870c92186bc674d65f62f2c/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The Bootlin i686 toolchain was already made available only on BR2_i386
with !BR2_x86_i486 && !BR2_x86_i586 && !BR2_x86_1000. However, this
was not sufficient as a few other architecture variants of BR2_i386
are "lower" than i686, and they need to be excluded as well.
Allows to fix:
http://autobuild.buildroot.net/results/56ac1a8fa5b34a9ca10eef98ae9fb090b8c762c4/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The current description of the Bootlin ARM toolchains allows them to
be selected for ARM big endian configurations, which obviously doesn't
work as these Bootlin ARM toolchains are little endian only.
We fix this by adding BR2_arm in the list of conditions for those
toolchains.
Will allow to fix:
http://autobuild.buildroot.net/results/7befbb686bb972016ba4e742976dcdb3fed1be11/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
We now support both MMU-enabled and MMU-less RISC-V 64-bit
configurations. However, the Bootlin toolchain for RISC-V 64-bit only
supports MMU-enabled configurations, but the current logic in
toolchain/toolchain-external/toolchain-external-bootlin/ does not take
this into account, and allows selecting the Booltin toolchain for
MMU-less RISC-V 64-bit configurations.
To fix this, the gen-bootlin-toolchains script is modified to add the
BR2_USE_MMU dependency to the description of the RISC-V 64-bit
toolchain.
However, the BR2_USE_MMU dependency was also added for glibc and musl
toolchains unconditionally, so to avoid duplicating the dependency, we
now only add it only if not already present in the list of
dependencies for this toolchain.
This will allow to fix:
http://autobuild.buildroot.net/results/d6aee9b275b1ec399aea59758ac8f69fdc5691fc/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The Config.in options created for each toolchain were properly taking
into account the !BR2_STATIC_LIBS dependency of glibc
toolchains. However, this dependency was not taken into account into
the main BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS
option. Consequently, if an architecture is only supported by glibc,
but BR2_STATIC_LIBS is enabled, the main "Bootlin toolchain" option
was visible... but with no selectable toolchain.
We fix this by making sure that
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS is only true for all
architectures supported, taking into account the fact that some
architectures can only be supported if !BR2_STATIC_LIBS, when the only
available C library is glibc.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Some constraints on a setup ended up with a plus sign in the path
for historical reasons and would then fail to match on the comparison
of the host/lib dir match. So, the =~ for bash can be augmented
with a double quote expansion to preserve the literal value of
the characters in the variable.
Example Path: /home/vagrant/test+buildroot/per-package
Signed-off-by: Charles Hardin <ckhardin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
pkg-stats currently uses the services from support/scripts/cpedb.py to
match the CPE identifiers of packages with the official CPE database.
Unfortunately, the cpedb.py code uses regular ElementTree parsing,
which involves loading the full XML tree into memory. This causes the
pkg-stats process to consume a huge amount of memory:
thomas 1310458 85.2 21.4 3708952 3450164 pts/5 R+ 16:04 0:33 | | \_ python3 ./support/scripts/pkg-stats
So, 3.7 GB of VSZ and 3.4 GB of RSS are used by the pkg-stats
process. This is causing the OOM killer to kick-in on machines with
relatively low memory.
This commit reimplements the XML parsing needed to do the CPE matching
directly in pkg-stats, using the XmlParser functionality of
ElementTree, also called "streaming parsing". Thanks to this, we never
load the entire XML tree in RAM, but only stream it through the
parser, and construct a very simple list of all CPE identifiers. The
max memory consumption of pkg-stats is now:
thomas 1317511 74.2 0.9 381104 152224 pts/5 R+ 16:08 0:17 | | \_ python3 ./support/scripts/pkg-stats
So, 381 MB of VSZ and 152 MB of RSS, which is obviously much better.
The JSON output of pkg-stats for the full package set, before and after
this commit, is exactly identical.
Now, one will probably wonder why this isn't directly changed in
cpedb.py. The reason is simple: cpedb.py is also used by
support/scripts/missing-cpe, which (for now) heavily relies on having
in memory the ElementTree objects, to re-generate a snippet of XML
that allows us to submit to NIST new CPE entries.
So, future work could include one of those two options:
(1) Re-integrate cpedb.py into missing-cpe directly, and live with
two different ways of processing the CPE database.
(2) Rewrite the missing-cpe logic to also be compatible with a
streaming parsing, which would allow this logic to be again
shared between pkg-stats and missing-cpe.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- add missing import of requests
- import CPEDB_URL from cpedb, instead of duplicating it
- fix flake8 errors
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Some upstream sites are very slow to respond, and the default timeout
of 300 seconds of the aiohttp.ClientSession() is too long. Let's
reduce it to 15 seconds.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This is useful when debugging/developing the pkg-stats script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This is useful when debugging/developing the pkg-stats script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This commit adds support for a new type of graph, showing the timeline
of a build. It shows, with one line per package, when each of this
package steps started/ended, and therefore allows to see the
sequencing of the package builds.
For a fully serialized build like we have today, this is not super
useful (except to show that everything is serialized), but it becomes
much more useful in the context of top-level parallel build.
We chose to order the graph by the time-of-configure, as it is the
closest to the actual cascade-style of a true dependency graph, which is
tiny bit more complex to achieve properly. The actual result still looks
pretty good.
The graph-build make target is extended to also generate this new
timeline graph.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- sort by start-of-configure time
- re-use existing colorsets (default or alternate)
- fix python2isms
- fix check-package
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The check_package_get_latest_version_by_distro() function analyzes the
data returned by release-monitoring.org. For two of our
packages (bento4 and qextserialport), release-monitoring.org returns
something that is a bit odd: it returns an entry with a
"stable_versions" field that contains an empty array. Our code was
ready to have or not have a "stable_versions" entry, but when it is
present, we assumed it was not an empty array. These two packages, for
some reason, break this assumption.
In order to solve this problem, this commit is more careful, and uses
the stable_versions field only if it exists and it has at least one
entry. The code is also reworked as a sequence of "if...elif...else"
to be more readable.
This fixes the following exception when running pkg-stats on the full
package set:
Task exception was never retrieved
future: <Task finished name='Task-10772' coro=<check_package_latest_version_get() done, defined at ./support/scripts/pkg-stats:532> exception=IndexError('list index out of range')>
Traceback (most recent call last):
File "./support/scripts/pkg-stats", line 535, in check_package_latest_version_get
if await check_package_get_latest_version_by_distro(session, pkg):
File "./support/scripts/pkg-stats", line 489, in check_package_get_latest_version_by_distro
version = data['stable_versions'][0] if 'stable_versions' in data else data['version'] if 'version' in data else None
IndexError: list index out of range
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: non-sequence tests as True]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 471ecea5ee (core/show-info: 'name' only applies to packages)
removed the 'name' field for rootfs (really, for non-package) entries,
thus breaking the pkg-stats processing.
We fix that by excluding any entry that has no 'name', on the assumption
that if it has no name, it is not a package.
Reported-by: Xogium on IRC
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This adds pep517(needed for flit-core to build itself) and flit python
package types.
We need to add an installer script and pass it appropriate options for
installing pep517 wheels generated by python-pypa-build during the
build stage. Unfortunately it seems pep517 does not support builds
without using the wheel format.
We also need to add a patch fixing the version parser in flit-core.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Arnout:
- fix indentation in pkg-python.mk (tabs, not spaces);
- use the new _CMD variables instead of duplicating the entire _CMDS
definitions;
- no need to filter dependencies (they're not self-referencing);
- _NEEDS_HOST_PYTHON no longer exists;
- host-python-pypa-build gets added to DEPENDENCIES automatically.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
... so we can catch regressions on check-package.
Update to the new docker image that was pushed after the previous
commit.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Some software decides based on uid/gid whether a user is a system or
normal (human) user, with different behaviour for those flavors (example
journald [2]).
So adding logic to create system-users is necessary, we take the now
common ranges from [1].
This extends the mkusers script to allow -2 for uid/gid, this argument
will take an identifier from the user range. All identifiers used up to
now should have been from the system range, so -1 is now interpreted as
a system user/group.
Note that after this commit, all the UIDs and GIDs that are created
automatically (with -1) will change. That means if there is peristent
data on an existing system that was created by such an automatic user,
it will suddenly belong to a different user. However, this could already
happen before: if a USERS line is added to a package, then other UIDs
may change as well.
Add system/user ranges as variables, and the argument for user/system
uid variable as well. Thus some magic constants could be removed, some
further occurences of -1 were replaced with equivalent logic. For
consistency, the existing MIN/MAX_UID/GID variables are renamed to
FIRST/LAST_USER_UID/GID.
Update the documentation with the new automatic ranges.
[1] - https://systemd.io/UIDS-GIDS/
[2] - https://www.freedesktop.org/software/systemd/man/journald.conf.html
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Arnout: use -1 for system users; refactor the changes a bit]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The x86-64-v4 toolchain assumes availability of AVX512, as per the
definition of the x86-64-v4 "standard".
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Following the merge of
d6ce2a1681 ("arch/Config.in.x86: add
option for -march=x86-64") and
eeace1cc13 ("arch/Config.in.x86: add support for
x86-64-v2, x86-64-v3, x86-64-v4"), bootlin.toolchains.com now provides
toolchains targetting the x86-64, x86-64-v2, x86-64-v3 and x86-64-v4
architecture variants.
This commits modifies gen-bootlin-toolchains to support these
toolchains. It should be noted that the description for the x86-64-v3
and x86-64-v4 toolchains are for now the same, as Buildroot doesn't
yet have the options to describe the extra features that x86-64-v4
expects to find on the hardware platform.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
support/scripts/pkg-stats:1171:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1175:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1179:8: E713 test for membership should be 'not in'
3 E713 test for membership should be 'not in'
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/1955772278
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
