Commit Graph

566 Commits

Author SHA1 Message Date
Peter Korsgaard
be914b97ad Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2022-09-11 09:57:08 +02:00
Arnout Vandecappelle
6c7061f3a8 support/scripts/pkg-stats: fix flake8 errors
Fixes flake8 errors:
support/scripts/pkg-stats:1013:133: E501 line too long (164 > 132 characters)
support/scripts/pkg-stats:1018:36: F541 f-string is missing placeholders
support/scripts/pkg-stats:1110:199: E261 at least two spaces before inline comment

For the first and the last one, we chose to split the long lines rather
than adding noqa: 501. Indeed, the long lines make it very unreadable,
and there are relatively natural places where the line can be broken.
Also split a line just below the second one in a similar way.

The f-string on 1018 doesn't need to be an f-string.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2022-08-27 23:02:38 +02:00
Thomas Petazzoni
d68628a538 support/scripts/pkg-stats: remove remaining double quote escaping
This is done either by switching to single quoted f-strings, triple
double quoted f-strings when needed, or simply single-quoted strings.

The renderer HTML is exactly identical before/after this commit.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
2022-08-24 13:36:12 +02:00
Thomas Petazzoni
98a3fba940 support/scripts/pkg-stats: remove useless escaping of double quotes
Within single-quoted f-strings, and within triple double quoted
strings, escaping all the double quotes is completely useless and
makes the code more difficult to read. Get rid of all this useless
escaping.

The renderer HTML is exactly identical before/after this commit.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
2022-08-24 13:35:49 +02:00
James Hilliard
fc429c4dc6 package/pkg-python: clean conflicting pep517 packages before install
The python installer package isn't able to overwrite files of packges
that already exist, this causes problems when doing a rebuild or
update without a full clean.

To fix this we can use functionality from importlib to identify and
remove any conflicting python package files before installation.

We also need to use internals from python-installer, as we want to use
the same logic as pyinstaller uses internally for getting the scheme so
that we ensure we clean the correct package scheme (we want it to be the
same as the one we're installing)

Fixes:
Traceback (most recent call last):
  File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
    main()
  File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
    install(
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
[yann.morin.1998@free.fr:
  - extend commit log about the use of the installer internals (the
    symbols prefixed with '_')
  - check path.files against explicitly None
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-08-22 11:54:25 +02:00
James Hilliard
d752f73b0c package/pkg-python: use pyinstaller.py for host python packages
The python installer cli isn't able to overwrite files of packages
that already exist, this causes problems when doing a rebuild or
update without a full clean.

Since we need to add functionality to our pyinstaller.py script to fix
this issue we must also use pyinstaller.py for host python packages.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-08-22 11:39:27 +02:00
Sen Hastings
ccc924ffb3 support/scripts/pkg-stats: add linebreak opportunity in CPE ID
This adds an optional linebreak at the vendor attribute in the CPE ID.

It should be noted this is purely for formatting/layout purposes
and does not actually insert any additional characters
(newline or otherwise) into the rendered text.

This means that even though the text renders across two lines,
copy-pasting will still yield one line of text.

example: https://sen-h.github.io/pkg-stats/c245575.html

see also: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/wbr

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-03 18:17:43 +02:00
Sen Hastings
559648a257 support/scripts/pkg-stats: re-implement the sortable_hint
This attempts to re-implement the "sortable_hint" feature without
relying on words. The column headers and CVE expand/contract buttons
change color and cursor style on hover.

If Javascript is enabled:

Just like [PATCH 1/3] more rules are applied to the
generated stylesheet before content is loaded.

If Javascript is disabled:

The headers stay pearly white. :-)

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-01 18:47:07 +02:00
Sen Hastings
d45b5377c1 support/scripts/pkg-stats: make cells with many CVEs collapsible
Sometimes a package can have a lot of CVEs.
Rather than have the CVE cell make a really tall row
(that means you have to scroll a bunch) this collapses the CVE
cell to a fixed size scrollable element with a
sticky button that lets you expand and collapse it.

If Javascript is enabled:

A stylesheet is generated and appended before content rendering,
amending the cells style to have a fixed height and overflow.

Also, the expand/contract button is unhidden.

This means the CVE cells are rendered in a collapsed state
instead of being rendered in an expanded state and then
slamming shut.

This avoids a "flash" and *helps* (vertically at least) manage CLS
(cumulative layout shift).

see: https://web.dev/cls/

If Javascript is disabled:

The cells stay fully open and the expand/contract button stays hidden.

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-08-01 18:47:07 +02:00
Yann E. MORIN
2a592f0b2a support/scripts: fix TABs/spaces in check-host-rpath
Commit c4e6d5c8be (core: implement per-package SDK and target)
introduced leading TABs in that otherwise-space-indented script.

Convert all to spaces, for homogeneity

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-07-30 18:15:08 +02:00
Thomas Petazzoni
c857b8ff0e Makefile, docs/manual, support, toolchain: remove Eclipse integration
Back many years ago, we developed an Eclipse plugin that simplified
the usage of Buildroot toolchains. Enabling the BR2_ECLIPSE_REGISTER=y
was registering the Buildroot toolchain into a special file in your
HOME folder that the Eclipse plugin would recognize to allow to
directly use the Buildroot cross-compiler.

This Eclipse plugin has not been maintained for years. The last commit
in the repository dates back from September 2017. Since then Eclipse
has moved on, and the plugin is no longer compatible with current
versions of Eclipse.

Also, Eclipse is probably no longer that widely used in the embedded
Linux space, as other more modern IDEs have become more popular.

All in all, it's time to say good bye to this Eclipse integration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-07-30 12:32:41 +02:00
Sen Hastings
76acf02c34 support/scripts/pkg-stats: fixed sortGrid() performance
sortGrid() has been rewritten to dynamically generate stylesheets with
explicit grid-row properties to re-order the rows, instead of removing
and reinserting the cells.
Performance *should* now be comperable to sorttable.js.

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-28 08:52:28 +02:00
Sen Hastings
786f8b4567 support/scripts/pkg-stats: fix duplicate package class names across rows
Like all good problems, disparate pieces work together to create
a "synergistically" hairy mess.

The sortGrid() overhaul highlighted a flaw in pkg-stats allowing
for duplicate package class names across rows.

As an example,

boot/barebox/barebox.mk and boot/barebox/barebox/barebox.mk
both get the classname ._barebox and so sortGrid() sticks them on
the same line giving a table with a vestigal row sticking out
of the right side like some kind of appendage.

Also I neglected to add a "_" to the current version column's cells
pkgname class so instead of "._pkgname" we had ".pkgname" and so
the cells were not collected properly as part of the row.

These issues explain the formatting weirdness.

package classnames are now ".path_to_package_makefile" without suffix
(.mk) (so ._boot_barebox_barebox and ._boot_barebox_barebox_barebox
instead of ._barebox) in order to guarantee uniqueness.

and what was *accidentally*
class="centered current_version data .barebox" is now
class="centered current_version data ._boot_barebox_barebox"
just like *all* the other cells in the row. :p

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-28 08:52:27 +02:00
Thomas Petazzoni
f1e3b2bb01 support/scripts/pkg-stats: add better checking in check_package_get_latest_version_by_guess()
The 'rtl8723ds' name, when queried from release-monitoring.org at
https://release-monitoring.org/api/projects/?pattern=rtl8723ds returns
one project, with one "stable_versions" array, which is empty. This
was not expected by the pkg-stats code, causing an exception:

Task exception was never retrieved
future: <Task finished name='Task-764' coro=<check_package_latest_version_get() done, defined at /home/thomas/projets/buildroot/./support/scripts/pkg-stats:558> exception=IndexError('list index out of range')>
Traceback (most recent call last):
  File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 566, in check_package_latest_version_get
    if await check_package_get_latest_version_by_guess(session, pkg):
  File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 544, in check_package_get_latest_version_by_guess
    projects[0]['stable_versions'][0],
IndexError: list index out of range

This commit therefore improves the checks done on the results received
from release-monitoring.org to avoid this issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 23:12:41 +02:00
Sen Hastings
e58e0d17c7 support/scripts/pkg-stats: updated copyright notice
Pretty straightforward. Adds my name to copyright notice.

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 23:12:41 +02:00
Sen Hastings
ff3acfe13a support/scripts/pkg-stats: fixed numeral in beginning of CSS class
CSS classes are generated for each package name for sorting purposes,
However some package names start with a number and this is not allowed.
(see https://www.w3.org/TR/CSS2/syndata.html#value-def-identifier)
Fix is to prepend a character to every class name such as "_".
so every ".package" is now "._package".

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 23:12:40 +02:00
Sen Hastings
af13345d43 support/scripts/pkg-stats: fixed CSS cascade
This fixes the .version-needs-update class being overridden by .correct class.

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 21:06:09 +02:00
Thomas Petazzoni
d353d30dee support/scripts/check-host-rpath: send readelf error output to oblivion
Somewhere between binutils 2.35 and 2.37, some functionality was
added in readelf to parse more DWARF information. Unfortunately, as
reported in binutils bug
28981 ("https://sourceware.org/bugzilla/show_bug.cgi?id=28981"), this
feature causes a number of fairly scary warnings to be displayed when
running readelf on binaries built with Clang, such as the pre-built
rustc and rustdoc binaries part of the host-rust-bin package. It
looks like this:

readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 2f in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 10b in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Bogus end-of-siblings marker detected at offset 10c in .debug_info section
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Further warnings about bogus end-of-sibling markers suppressed
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x23
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: Unrecognized form: 0x22
readelf: /home/thomas/projets/buildroot/output/host/bin/rustc: Warning: DIE at offset 0x1da refers to abbreviation number 5827 which does not exist

These warnings are caused by the readelf calls done by the
support/scripts/check-host-rpath script. The annoying thing is that
once host-rust-bin has been installed in $(HOST_DIR), this warning
appears after the installation of every single host package, because
support/scripts/check-host-rpath rescans all binaries every time.

To avoid showing those scary warnings, this commit sends the error
output of readelf to /dev/null.

Of course, it would be nicer to only filter out those warnings, but
filtering the error output without merging the error output into the
standard output is tricky, so let's keep things simple. If there is
really an error, readelf will abort.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-07-27 18:36:18 +02:00
Romain Naour
42f7f451ec support/scripts/boot-qemu-image.py: increase the timeout by 10
As for the Buildroot testsuite, multiply every emulator timeout by 10
to avoid sporadic failures in elastic runners.

qemu_arm_vexpress_tz_defconfig tested locally with sucess.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 15:56:57 +02:00
Arnout Vandecappelle (Essensium/Mind)
90932b407c toolchain: invert glibc <-> !static dependency
Currently, glibc depends on !BR2_STATIC_LIBS in all the toolchain
variants.

However, for some architectures, glibc is the only supported libc. In
commit 3b3105328e ("Config.in: only
allow BR2_STATIC_LIBS on supported libc/arch"), we implemented a fix
to avoid configurations were BR2_STATIC_LIBS=y with an architecture
already supported by glibc, because these configurations are
impossible. This commit 3b3105328e
prevents from selecting BR2_STATIC_LIBS=y when the C library used for
the internal toolchain backend is glibc.

However, it introduces a discrepency between how this topic is handled
for internal and external toolchains:

 - For internal toolchains, we prevent BR2_STATIC_LIBS=y if glibc is
   chosen.

 - For external toolchains, we allow BR2_STATIC_LIBS=y in all cases,
   and it's each glibc toolchain that has !BR2_STATIC_LIBS

This commit addresses this discrepency by preventing BR2_STATIC_LIBS=y
if glibc is chosen in all cases.

Thanks to this, we can remove the !BR2_STATIC_LIBS dependency on both
the glibc package, and all glibc external toolchains.

Fixes: https://bugs.busybox.net/show_bug.cgi?id=14256

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: update to master, fix the gen-bootlin-toolchains script, add
a comment in the static/shared choice to indicate that static is
supported only with uclibc or musl]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-27 11:11:19 +02:00
Heiko Thiery
28973f28ac support/scripts/pkg-stats: tweak infras field when running with -c
When we use the statistics output to generate a CVE/CPE customer
report showing whether a product is affected by CVEs, we are primarily
interested in whether they are relevant to the target
system. Currently we cannot see if the package is configured for the
build (infra==host) and/or the target system (infra==target).

Therefore this commit extends the pkg-stats script to leverage the
information available in "make show-info" output to tweak the list of
package infrastructures for each package. Thanks to this commit, the
script now has a more consistent behavior:

 * When pkg-stats is run without -c, i.e without a defined Buildroot
   configuration, it continues to operate as it did, i.e it lists all
   package infrastructures supported by the package (such as autotools
   host+target, or kconfig target, etc.)

 * When pkg-stats is run with -c, i.e with a defined Buildroot
   configuration which defines the list of packages that should be
   considered, then for each package it now lists only the package
   infrastructures used by the package in that current
   configuration. For example if you have a package with a host and
   target variant, but only the host variant is used in your
   configuration, now the pkg-stats output will only say that the host
   variant of this package is used;

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: pretty much rework the entire implementation and how the
result is presented.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-07-25 19:08:52 +02:00
Arnout Vandecappelle (Essensium/Mind)
bf178754ef support/scripts/pkg-stats: fix flake8 errors
Commit f1bcb2a45c introduced a number of
flake8 errors. Fix these by:
- adding noqa to the multi-line string containing tabs;
- replacing other tabs with spaces;
- removing space after opening parenthesis;
- splitting the long lines.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-07-23 19:47:11 +02:00
Sen Hastings
f1bcb2a45c support/scripts/pkg-stats: migrate to CSS grid and inline javascript
This migrates pkg-stats.html from html tables to CSS grid, allowing
the use of newer, simpler javascript that is short enough to be
inlined, instead of relying on externally hosted javascript.

Javascript sorting function was rewritten from scratch in ~55 lines,
short enough to be inlined directly in the html.

Tables were redone in CSS grid, but with care taken to mimic existing
"look and feel" of prevous implementation, albeit with slightly
better responsive behavior and default styling characteristics.

Column labels are now "sticky" and stay stuck to the top of the
viewport as you scroll down the page.

Also, css was rewritten in fewer lines and table elements were changed
to divs (for grid support).

Other small misc fixes include quoted hrefs and document language
declarations to make the w3c html validator happy.

Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-07-23 18:09:52 +02:00
Vincent Fazio
3fed424566 package/python3: use the provided pyc compiler
Previously, we used support/scripts/pycompile.py to generate the pyc
files for the python libraries.

While the script worked, it did not follow the PEP 3147 layout
requirements for py+pyc deployments.

Now, use the package's own compileall.py script. This will follow
PEP 3147 guidelines. It also supports "legacy" pyc only deployments as
described here:
  https://peps.python.org/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports

With this change, we no longer need to hack support for side-by-side pyc
files because files will be deployed as appropriate.

This also has the added benefit of not requiring python3 on the host to
build host-python3.

Fixes: #14911

Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
  - build-tested in a python-less environment
  - build+run-tested with the runtime-test infra
]
Tested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-07-23 17:23:57 +02:00
Nicolas POIROT
4de196a1ce support/scripts/br2-external: add kernel extensions
Buildroot documentation section 9.2.1.6 "Additional kernel extensions"
indicates support for kernel extensions defined in external buildroot
trees but unfortunately, there didn't seem to be any support in
br2-external script.

This patch copies 'init' code support to include external kernel
extensions defined in 'linux' dir at the br2-external root directory as
explained in documentation.

Signed-off-by: Nicolas POIROT <ni.poirot@laposte.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-06-22 22:22:30 +02:00
Thomas Petazzoni
3008505865 support/scripts/gen-bootlin-toolchains: add support for ARMv7 big endian toolchain
This Bootlin toolchain has been available for a while, but was not
supported until now.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 17:28:30 +02:00
Thomas Petazzoni
db37e75aa3 support/scripts/gen-bootlin-toolchains: drop "mips64" architecture variant
These toolchains have been marked obsolete by toolchains.bootlin.com
as they are replaced by the mips64-n32 toolchains. See commit:

  121e78806b

Will allow to fix:

  http://autobuild.buildroot.net/results/d13b35ba5a0f68f72e6592bdd9218b625a3c6554/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 17:28:30 +02:00
Thomas Petazzoni
3c11e6f3c8 support/scripts/gen-bootlin-toolchains: check that toolchains exists for a certain arch
The "arches" dict lists the architectures for which Bootlin toolchains
are available, along with the corresponding Buildroot conditions for
those architecture variants.

However, there is nothing that checks that such architectures really
have Bootlin toolchains available. Even if no toolchain is available,
the architecture is considered as support, and will be listed in the
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS option, making menuconfig
believe that some Bootlin toolchain is available for the selected
architecture variant.

This is currently the case with the "mips64" architecture (as
identified by the Bootlin toolchains project). Such toolchains have
been made obsolete in toolchains.bootlin.com and replaced by the
mips64-n32 toolchains. But "mips64" is still listed in "arches",
causing this architecture to be considered as having Bootlin
toolchains available.

To avoid this to happen in the future, this commit adds a check that
verifies that an architecture listed in "arches" really has at least
one toolchain available.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 17:28:30 +02:00
Thomas Petazzoni
a5599d727e support/scripts/gen-bootlin-toolchains: make use of BR2_TOOLCHAIN_EXTERNAL_HAS_NO_GDBSERVER
The microblazeel, microblazebe and openrisc toolchains need to select
this option as they don't have any gdb/gdbserver.

Helps in fixing:

  http://autobuild.buildroot.net/results/6315ef7b66ee4ae8f870c92186bc674d65f62f2c/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 17:28:30 +02:00
Thomas Petazzoni
26dee54e42 support/scripts/gen-bootlin-toolchains: adjust dependencies of i686 toolchains
The Bootlin i686 toolchain was already made available only on BR2_i386
with !BR2_x86_i486 && !BR2_x86_i586 && !BR2_x86_1000. However, this
was not sufficient as a few other architecture variants of BR2_i386
are "lower" than i686, and they need to be excluded as well.

Allows to fix:

  http://autobuild.buildroot.net/results/56ac1a8fa5b34a9ca10eef98ae9fb090b8c762c4/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 16:49:32 +02:00
Thomas Petazzoni
f2be0bb278 support/scripts/gen-bootlin-toolchains: improve dependencies of ARM toolchains
The current description of the Bootlin ARM toolchains allows them to
be selected for ARM big endian configurations, which obviously doesn't
work as these Bootlin ARM toolchains are little endian only.

We fix this by adding BR2_arm in the list of conditions for those
toolchains.

Will allow to fix:

  http://autobuild.buildroot.net/results/7befbb686bb972016ba4e742976dcdb3fed1be11/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 16:49:27 +02:00
Thomas Petazzoni
1f4bba5967 support/scripts/gen-bootlin-toolchains: fix RISC-V 64-bit toolchain description
We now support both MMU-enabled and MMU-less RISC-V 64-bit
configurations. However, the Bootlin toolchain for RISC-V 64-bit only
supports MMU-enabled configurations, but the current logic in
toolchain/toolchain-external/toolchain-external-bootlin/ does not take
this into account, and allows selecting the Booltin toolchain for
MMU-less RISC-V 64-bit configurations.

To fix this, the gen-bootlin-toolchains script is modified to add the
BR2_USE_MMU dependency to the description of the RISC-V 64-bit
toolchain.

However, the BR2_USE_MMU dependency was also added for glibc and musl
toolchains unconditionally, so to avoid duplicating the dependency, we
now only add it only if not already present in the list of
dependencies for this toolchain.

This will allow to fix:

  http://autobuild.buildroot.net/results/d6aee9b275b1ec399aea59758ac8f69fdc5691fc/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-06-14 16:49:22 +02:00
Thomas Petazzoni
fc63ec9292 support/scripts/gen-bootlin-toolchains: properly take into account !BR2_STATIC_LIBS for glibc toolchains
The Config.in options created for each toolchain were properly taking
into account the !BR2_STATIC_LIBS dependency of glibc
toolchains. However, this dependency was not taken into account into
the main BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS
option. Consequently, if an architecture is only supported by glibc,
but BR2_STATIC_LIBS is enabled, the main "Bootlin toolchain" option
was visible... but with no selectable toolchain.

We fix this by making sure that
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARCH_SUPPORTS is only true for all
architectures supported, taking into account the fact that some
architectures can only be supported if !BR2_STATIC_LIBS, when the only
available C library is glibc.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-06-06 15:15:01 +02:00
Charles Hardin
2a9ef1f572 support/scripts: handle paths with regex characters in per-package
Some constraints on a setup ended up with a plus sign in the path
for historical reasons and would then fail to match on the comparison
of the host/lib dir match. So, the =~ for bash can be augmented
with a double quote expansion to preserve the literal value of
the characters in the variable.

Example Path: /home/vagrant/test+buildroot/per-package

Signed-off-by: Charles Hardin <ckhardin@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-06-04 09:55:34 +02:00
Thomas Petazzoni
bd1798ad95 support/scripts/pkg-stats: reimplement CPE parsing in pkg-stats
pkg-stats currently uses the services from support/scripts/cpedb.py to
match the CPE identifiers of packages with the official CPE database.

Unfortunately, the cpedb.py code uses regular ElementTree parsing,
which involves loading the full XML tree into memory. This causes the
pkg-stats process to consume a huge amount of memory:

thomas   1310458 85.2 21.4 3708952 3450164 pts/5 R+   16:04   0:33  |   |   \_ python3 ./support/scripts/pkg-stats

So, 3.7 GB of VSZ and 3.4 GB of RSS are used by the pkg-stats
process. This is causing the OOM killer to kick-in on machines with
relatively low memory.

This commit reimplements the XML parsing needed to do the CPE matching
directly in pkg-stats, using the XmlParser functionality of
ElementTree, also called "streaming parsing". Thanks to this, we never
load the entire XML tree in RAM, but only stream it through the
parser, and construct a very simple list of all CPE identifiers. The
max memory consumption of pkg-stats is now:

thomas   1317511 74.2  0.9 381104 152224 pts/5   R+   16:08   0:17  |   |   \_ python3 ./support/scripts/pkg-stats

So, 381 MB of VSZ and 152 MB of RSS, which is obviously much better.

The JSON output of pkg-stats for the full package set, before and after
this commit, is exactly identical.

Now, one will probably wonder why this isn't directly changed in
cpedb.py. The reason is simple: cpedb.py is also used by
support/scripts/missing-cpe, which (for now) heavily relies on having
in memory the ElementTree objects, to re-generate a snippet of XML
that allows us to submit to NIST new CPE entries.

So, future work could include one of those two options:

 (1) Re-integrate cpedb.py into missing-cpe directly, and live with
     two different ways of processing the CPE database.

 (2) Rewrite the missing-cpe logic to also be compatible with a
     streaming parsing, which would allow this logic to be again
     shared between pkg-stats and missing-cpe.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - add missing import of requests
  - import CPEDB_URL from cpedb, instead of duplicating it
  - fix flake8 errors
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-04-02 19:14:17 +02:00
Thomas Petazzoni
387c496b98 support/scripts/pkg-stats: add a timeout on HTTP requests for upstream URLs
Some upstream sites are very slow to respond, and the default timeout
of 300 seconds of the aiohttp.ClientSession() is too long. Let's
reduce it to 15 seconds.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-04-02 16:36:09 +02:00
Thomas Petazzoni
1adcf56598 support/scripts/pkg-stats: allow disabling package warnings retrieval
This is useful when debugging/developing the pkg-stats script.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-04-02 16:34:37 +02:00
Thomas Petazzoni
cd8a576c6d support/scripts/pkg-stats: allow disabling CPE matching
This is useful when debugging/developing the pkg-stats script.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-04-02 16:34:07 +02:00
Thomas Petazzoni
5e8b01afd5 support/scripts/graph-build-time: add support for timeline graphing
This commit adds support for a new type of graph, showing the timeline
of a build. It shows, with one line per package, when each of this
package steps started/ended, and therefore allows to see the
sequencing of the package builds.

For a fully serialized build like we have today, this is not super
useful (except to show that everything is serialized), but it becomes
much more useful in the context of top-level parallel build.

We chose to order the graph by the time-of-configure, as it is the
closest to the actual cascade-style of a true dependency graph, which is
tiny bit more complex to achieve properly. The actual result still looks
pretty good.

The graph-build make target is extended to also generate this new
timeline graph.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - sort by start-of-configure time
  - re-use existing colorsets (default or alternate)
  - fix python2isms
  - fix check-package
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-03-20 23:52:24 +01:00
Thomas Petazzoni
c72f3f2b43 support/scripts/pkg-stats: strengthen version check in check_package_get_latest_version_by_distro()
The check_package_get_latest_version_by_distro() function analyzes the
data returned by release-monitoring.org. For two of our
packages (bento4 and qextserialport), release-monitoring.org returns
something that is a bit odd: it returns an entry with a
"stable_versions" field that contains an empty array. Our code was
ready to have or not have a "stable_versions" entry, but when it is
present, we assumed it was not an empty array. These two packages, for
some reason, break this assumption.

In order to solve this problem, this commit is more careful, and uses
the stable_versions field only if it exists and it has at least one
entry. The code is also reworked as a sequence of "if...elif...else"
to be more readable.

This fixes the following exception when running pkg-stats on the full
package set:

Task exception was never retrieved
future: <Task finished name='Task-10772' coro=<check_package_latest_version_get() done, defined at ./support/scripts/pkg-stats:532> exception=IndexError('list index out of range')>
Traceback (most recent call last):
  File "./support/scripts/pkg-stats", line 535, in check_package_latest_version_get
    if await check_package_get_latest_version_by_distro(session, pkg):
  File "./support/scripts/pkg-stats", line 489, in check_package_get_latest_version_by_distro
    version = data['stable_versions'][0] if 'stable_versions' in data else data['version'] if 'version' in data else None
IndexError: list index out of range

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: non-sequence tests as True]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-03-16 23:04:46 +01:00
Romain Naour
249ce1c584 support/scripts/gen-bootlin-toolchains: add s390x toolchain
s390x support has been added since toolchain-builder release
2021.05 [1].

[1] 4bbb52a6a3

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-03-10 22:09:22 +01:00
Peter Korsgaard
d0064df6bd Merge branch 'next' 2022-03-09 14:23:07 +01:00
Yann E. MORIN
0319170f50 support/pkg-stats: fix regression after cleanup in show-info
Commit 471ecea5ee (core/show-info: 'name' only applies to packages)
removed the 'name' field for rootfs (really, for non-package) entries,
thus breaking the pkg-stats processing.

We fix that by excluding any entry that has no 'name', on the assumption
that if it has no name, it is not a package.

Reported-by: Xogium on IRC
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-16 22:38:09 +01:00
James Hilliard
29a12eb86a package/python-flit-core: new package/infrastructure
This adds pep517(needed for flit-core to build itself) and flit python
package types.

We need to add an installer script and pass it appropriate options for
installing pep517 wheels generated by python-pypa-build during the
build stage. Unfortunately it seems pep517 does not support builds
without using the wheel format.

We also need to add a patch fixing the version parser in flit-core.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Arnout:
 - fix indentation in pkg-python.mk (tabs, not spaces);
 - use the new _CMD variables instead of duplicating the entire _CMDS
   definitions;
 - no need to filter dependencies (they're not self-referencing);
 - _NEEDS_HOST_PYTHON no longer exists;
 - host-python-pypa-build gets added to DEPENDENCIES automatically.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-16 22:16:43 +01:00
Ricardo Martincoski
fcceee0b2e utils/checkpackagelib: run unit tests on GitLab CI
... so we can catch regressions on check-package.

Update to the new docker image that was pushed after the previous
commit.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-06 15:35:19 +01:00
Norbert Lange
41ea61d59c support/scripts/mkusers: allow option for system uid/gid
Some software decides based on uid/gid whether a user is a system or
normal (human) user, with different behaviour for those flavors (example
journald [2]).

So adding logic to create system-users is necessary, we take the now
common ranges from [1].

This extends the mkusers script to allow -2 for uid/gid, this argument
will take an identifier from the user range. All identifiers used up to
now should have been from the system range, so -1 is now interpreted as
a system user/group.

Note that after this commit, all the UIDs and GIDs that are created
automatically (with -1) will change. That means if there is peristent
data on an existing system that was created by such an automatic user,
it will suddenly belong to a different user. However, this could already
happen before: if a USERS line is added to a package, then other UIDs
may change as well.

Add system/user ranges as variables, and the argument for user/system
uid variable as well. Thus some magic constants could be removed, some
further occurences of -1 were replaced with equivalent logic. For
consistency, the existing MIN/MAX_UID/GID variables are renamed to
FIRST/LAST_USER_UID/GID.

Update the documentation with the new automatic ranges.

[1] - https://systemd.io/UIDS-GIDS/
[2] - https://www.freedesktop.org/software/systemd/man/journald.conf.html

Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Arnout: use -1 for system users; refactor the changes a bit]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-02-05 23:09:31 +01:00
Thomas Petazzoni
7ed58f94ee support/scripts/gen-bootlin-toolchains: add missing BR2_X86_CPU_HAS_AVX512 condition for x86-64-v4
The x86-64-v4 toolchain assumes availability of AVX512, as per the
definition of the x86-64-v4 "standard".

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-01-25 08:41:19 +01:00
Thomas Petazzoni
93cb793181 support/scripts/gen-bootlin-toolchains: add support for new x86-64 toolchains
Following the merge of
d6ce2a1681 ("arch/Config.in.x86: add
option for -march=x86-64") and
eeace1cc13 ("arch/Config.in.x86: add support for
x86-64-v2, x86-64-v3, x86-64-v4"), bootlin.toolchains.com now provides
toolchains targetting the x86-64, x86-64-v2, x86-64-v3 and x86-64-v4
architecture variants.

This commits modifies gen-bootlin-toolchains to support these
toolchains. It should be noted that the description for the x86-64-v3
and x86-64-v4 toolchains are for now the same, as Buildroot doesn't
yet have the options to describe the extra features that x86-64-v4
expects to find on the hardware platform.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2022-01-25 08:40:59 +01:00
Arnout Vandecappelle (Essensium/Mind)
02e679d8bf support/scripts/pkg-stats: fix flake8 errors
support/scripts/pkg-stats:1171:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1175:8: E713 test for membership should be 'not in'
support/scripts/pkg-stats:1179:8: E713 test for membership should be 'not in'
3     E713 test for membership should be 'not in'

Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/1955772278

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2022-01-09 21:00:56 +01:00
Thomas Petazzoni
b102352b62 support/scripts/pkg-stats: add --disable option
When debugging pkg-stats, it's quite useful to be able to disable some
features that are quite long (checking upstream URL, checking latest
version, checking CVE). This commit adds a --disable option, which can
take a comma-separated list of features to disable, such as:

     ./support/scripts/pkg-stats --disable url,upstream

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2022-01-09 17:31:55 +01:00