Drop override system locale patch which is now upstream.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Now that host-python2 is gone, it makes sense to unconditionally install the
symlinks for the host, similar to how it is done for the target.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Refresh 0032-configure.ac-fixup-CC-print-multiarch-output-for-mus.patch now
that the MULTIARCH logic got updated by:
7e951f356e
Update license hash after a change of copyright years:
35955e4ade
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fix the following build failure raised since bump to version 3.10.1 in
commit 25b1fc2898:
/home/buildroot/autobuild/instance-3/output-1/build/python3-3.10.1/Modules/_hashopenssl.c:244:22: error: implicit declaration of function 'EVP_blake2s256'; did you mean 'LN_blake2s256'? [-Werror=implicit-function-declaration]
244 | digest = EVP_blake2s256();
| ^~~~~~~~~~~~~~
| LN_blake2s256
Fixes:
- http://autobuild.buildroot.org/results/9112571b75aebb0ba5032ef1b16226d9848f5184
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
- Remove 0013-Add-an-option-to-disable-installation-of-test-module.patch as
it is now upstreamed.
- Refactor and rename all other patches as necessary.
Signed-off-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Drop 0030-Fix-cross-compiling-the-uuid-module.patch as the patched code has
been reworked upstream and python3 is built with --disable-uuid:
91a51c5ffc
Rework 0033-configure.ac-fixup-CC-print-multiarch-output-for-mus.patch as
the MULTIARCH code is now conditional on !darwin:
9901d153c2
Refresh and renumber remaining patches.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fix the following build failure raised since commit
21a56f9edb which wrongly removed
the select on BR2_PACKAGE_BZIP2 from BR2_PACKAGE_PYTHON3_BZIP2:
Makefile:587: *** bzip2 is in the dependency chain of python3 that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in. Stop.
Fixes:
- http://autobuild.buildroot.org/results/041a2f9c536fce1a1acf6da0464855e4f1e1604f
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The bzip2 module is needed to build the host variant of NodeJS 14.
Signed-off-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fixes the following security issues:
- bpo-42278: Replaced usage of tempfile.mktemp() with TemporaryDirectory to
avoid a potential race condition.
- bpo-41180: Add auditing events to the marshal module, and stop raising
code.__init__ events for every unmarshalled code object. Directly
instantiated code objects will continue to raise an event, and audit event
handlers should inspect or collect the raw marshal data. This reduces a
significant performance overhead when loading from .pyc files.
- bpo-44394: Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to
get the fix for the CVE-2013-0340 “Billion Laughs” vulnerability. This
copy is most used on Windows and macOS.
- bpo-43124: Made the internal putcmd function in smtplib sanitize input for
presence of \r and \n characters to avoid (unlikely) command injection.
https://www.python.org/downloads/release/python-397/
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Highly parallel host-python3 builds sometimes fail with:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
worker_sentinels = [p.sentinel for p in self.processes.values()]
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration
During the compile_all.py step of host-python3. This issue is reported
upstream at https://bugs.python.org/issue43498, and while not yet
fixed upstream, a PR was proposed with a possible fix for it. Seems
the PR seems reasonable, let's give it a chance and see if it improves
the situation.
Hopefully Fixes:
http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Other changes:
- Rename 0034-lib-crypt-uClibc-ng-doesn-t-set-errno-when-encryptio.patch to
0035-lib-crypt-uClibc-ng-doesn-t-set-errno-when-encryptio.patch, as to not
overlap with 0034-Add-an-option-to-disable-the-berkeleydb-module.patch
Signed-off-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes the following security issues:
- bpo-43434: Creating a sqlite3.Connection object now also produces a
sqlite3.connect auditing event. Previously this event was only produced
by sqlite3.connect() calls. Patch by Erlend E. Aasland.
- bpo-43882: The presence of newline or tab characters in parts of a URL
could allow some forms of attacks.
Following the controlling specification for URLs defined by WHATWG
urllib.parse() now removes ASCII newlines and tabs from URLs, preventing
such attacks.
- bpo-43472: Ensures interpreter-level audit hooks receive the
cpython.PyInterpreterState_New event when called through the
_xxsubinterpreters module.
- bpo-36384: ipaddress module no longer accepts any leading zeros in IPv4
address strings. Leading zeros are ambiguous and interpreted as octal
notation by some libraries. For example the legacy function
socket.inet_aton() treats leading zeros as octal notatation. glibc
implementation of modern inet_pton() does not accept any leading zeros.
For a while the ipaddress module used to accept ambiguous leading zeros.
- bpo-43075: Fix Regular Expression Denial of Service (ReDoS) vulnerability
in urllib.request.AbstractBasicAuthHandler. The ReDoS-vulnerable regex
has quadratic worst-case complexity and it allows cause a denial of
service when identifying crafted invalid RFCs. This ReDoS issue is on the
client side and needs remote attackers to control the HTTP server.
- bpo-42800: Audit hooks are now fired for frame.f_code, traceback.tb_frame,
and generator code/frame attribute access.
https://www.python.org/downloads/release/python-395/
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following security issues:
- bpo-42988: CVE-2021-3426: Remove the getfile feature of the pydoc module
which could be abused to read arbitrary files on the disk (directory
traversal vulnerability). Moreover, even source code of Python modules
can contain sensitive data like passwords. Vulnerability reported by
David Schwörer.
- bpo-43285: ftplib no longer trusts the IP address value returned from the
server in response to the PASV command by default. This prevents a
malicious FTP server from using the response to probe IPv4 address and
port combinations on the client network.
Code that requires the former vulnerable behavior may set a
trust_server_pasv_ipv4_address attribute on their ftplib.FTP instances to
True to re-enable it.
- bpo-43439: Add audit hooks for gc.get_objects(), gc.get_referrers() and
gc.get_referents(). Patch by Pablo Galindo.
Note: 3.9.3 was recalled due to introducing unintentional ABI
incompatibility, and fixes re-released as 3.9.4:
https://www.python.org/downloads/release/python-394/
Add host-autoreconf-archive, as it is needed for autoreconf since:
064bc07f24
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following security issue:
- CVE-2021-23336: urllib.parse.parse_qsl(): Web cache poisoning - `; ` as a
query args separator
https://bugs.python.org/issue42967
And fixes a number of issues. For details, see the changelog:
https://docs.python.org/release/3.9.2/whatsnew/changelog.html
Drop the now upstreamed security patch and update the license hash for a
change of copyright year:
-2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation;
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Python Software Foundation;
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes the following security issue:
- CVE-2021-3177: Python 3.x through 3.9.1 has a buffer overflow in
PyCArg_repr in _ctypes/callproc.c, which may lead to remote code execution
in certain Python applications that accept floating-point numbers as
untrusted input, as demonstrated by a 1e300 argument to
c_double.from_param. This occurs because sprintf is used unsafely.
For details, see the advisory:
https://python-security.readthedocs.io/vuln/ctypes-buffer-overflow-pycarg_repr.html
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This rebases the remaining patches for python 3.9.0 not included in
f26ce57760.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Minimal rebase of patches on 3.9.0.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Contains a number of bugfixes. For details, see the changelog:
https://docs.python.org/release/3.8.6/whatsnew/changelog.html#changelog
Update the license hash for the addition of a note stating that the examples
and documentation is now dual licensed under the PSF and a Zero-Clause BSD
license since:
9fef7c54a0
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Buildroot generates Linux systems, so there is no need to have
MS Windows executables in there.
This reduces the target filesystem size by about 2MB:
$ du -hcs usr/lib/python3.8/distutils/command/wininst-*.exe
60K usr/lib/python3.8/distutils/command/wininst-6.0.exe
64K usr/lib/python3.8/distutils/command/wininst-7.1.exe
60K usr/lib/python3.8/distutils/command/wininst-8.0.exe
192K usr/lib/python3.8/distutils/command/wininst-9.0.exe
220K usr/lib/python3.8/distutils/command/wininst-9.0-amd64.exe
188K usr/lib/python3.8/distutils/command/wininst-10.0.exe
220K usr/lib/python3.8/distutils/command/wininst-10.0-amd64.exe
448K usr/lib/python3.8/distutils/command/wininst-14.0.exe
576K usr/lib/python3.8/distutils/command/wininst-14.0-amd64.exe
2,0M total
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a new option that prints the (runtime) path of compiled .py files
when VERBOSE=1 is set.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.
By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.
This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.
Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.
Rework the script to call py_compile.compile() directly with pertinent
options:
- The script now has a new --strip-root argument. This argument is
optional but will always be specified when compiling py files in
buildroot.
- All other (non-optional) arguments are folders in which all
"importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
file is computed and encoded into the compiled .pyc.
No need to change directory before running the script anymore.
The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.
Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).
Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.
So we just drop support for --force, and instead we always byte-compile.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
- always byte-compile
- drop --force
- expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Bugfix release, fixing a number of issues. For details, see the
announcement:
https://docs.python.org/release/3.8.2/whatsnew/changelog.html#python-3-8-2-final
Adjust the spacing in the hash file and update the hash of the license file
for a change in copyright years:
-2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation;
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation;
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With per-package directory support, Python external modules are
causing a problem: the _sysconfigdata.py module installed by the
Python interpreter contains a number of paths that are relative to the
current package per-package directory, i.e python or python3. For
example:
'BLDSHARED': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-gcc -shared',
'CC': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-gcc',
'CXX': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-g++',
etc.
These paths are problematic, because it means that the wrong compiler
gets used when building external Python modules: instead of using the
compiler from the external Python module per-package host directory,
it uses the one from the 'python' or 'python3' per-package host
directory. Due to this, any native dependency needed by the external
Python module is not found, even though it is properly present in the
current package per-package directory.
Of course, the problem occurs with both target Python modules and host
Python modules.
To fix this, we simply rewrite those paths in _sysconfigdata.py before
building a Python package.
Interestingly, until now, the _sysconfidata.py that was used during
the build was the one from $(TARGET_DIR), which is a bit unusual: it
is more common to use files from $(STAGING_DIR) during the build
process. So this commit changes the PYTHON_PATH and PYTHON3_PATH
variables so that they point to $(STAGING_DIR), which makes the
_sysconfigdata.py fixup in $(STAGING_DIR) effective.
Fixes:
http://autobuild.buildroot.net/results/a24b0555fd4261b50dc3986635c30717d9cbe764/ (python-psycopg2)
http://autobuild.buildroot.net/results/080fa893e1b0e7a8c8a31ac1c98eb8871b97264d/ (python-alsaaudio)
http://autobuild.buildroot.net/results/79bc070f98d6d9d8ef78df12b248cdc7d0e405c3/ (python-lxml)
and many more Python packages that use native code with a native library
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/3b6/3b6280b0b7a9634b747db2865b21c6266007c725/
The PYTHON_KEEP_PY_FILES global variable conflicts with the per-package
<pkg>_KEEP_PY_FILES variable for the python package, causing make to
complain:
package/zlib/zlib.mk:7: *** Recursive variable 'PYTHON_KEEP_PY_FILES' references itself (eventually). Stop.
As a workaround, rename the global variable to KEEP_PYTHON_PY_FILES so it
cannot conflict with the per-package variable.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files
from the system, as they have all been byte-compiled into their .pyc
variants.
However, it turns out that some packages (e.g: OpenCV) do some funky
things with a few .py files: they pass them through Python's
execfile() facility, which only works with .py files and not .pyc
files. It is used by OpenCV for example to read two small
configuration files.
In order to support such use cases, this commit introduces a very
simple mechanism by which packages can exclude some path patterns from
the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is
collected into a global PYTHON_KEEP_PY_FILES variable, then used by
the python/python3 target-finalize hooks.
This variable is intentionally not documented, this is really a hack
that we ideally would like to see go away, and we'd rather not see its
usage spread too much.
This is necessary to be able to fix bug #12171.
[Peter: check if PYTHON_KEEP_PY_FILES contains non-white space]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some python scripts may be ran in the custom scripts a user can define
in the config. Allow the user to enable host-python3 explicitly.
If any of those require ssl, they will fail with no possible fix.
Add an option to enable openssl as well. This is made optional because
openssl significantly increases the build time.
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
python3 nowadays appends the triplet to the config-<version>m directory:
echo target/usr/lib/python3.7/config-*
target/usr/lib/python3.7/config-3.7m-powerpc-linux-gnu
Likewise, there is no longer a pyconfig.h:
ls target/usr/lib/python3.7/config-3.7m-powerpc-linux-gnu
config.c config.c.in install-sh libpython3.7m.a Makefile
makesetup python-config.py python.o Setup Setup.local
So adjust the removal logic to match. Use a wildcard rather than
$GNU_TARGET_NAME as buildroot and python3's idea of the triplet doesn't
always match (E.G. for musl/uclibc).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fixes:
http://autobuild.buildroot.net/results/cb4/cb49c539501342e45cbe5ade82e588fcdf51f05b
GCC commit 6834b83784dcf0364eb820e8 (multiarch support for non-glibc linux
systems), which is part of GCC 8+, changed the multiarch logic to use
$arch-linux-musl / $arch-linux-uclibc rather than $arch-linux-gnu.
This then causes the python3 configure script to error out:
checking for the platform triplet based on compiler characteristics... powerpc-linux-gnu
configure: error: internal configure error for the platform triplet, please file a bug report
http://autobuild.buildroot.net/results/cb4/cb49c539501342e45cbe5ade82e588fcdf51f05b
As it requires that the --print-multiarch output (if not empty) matches the
deduced triplet (which always uses -linux-gnu).
It isn't quite clear why --print-multiarch returns something for a
non-multiarch toolchain on some architectures (E.G. PowerPC), but as a
workaround, add a patch to rewrite the --print-multiarch output to match
older GCC versions to keep the configure script happy.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fixes the following security issues:
- bpo-37463: ssl.match_hostname() no longer accepts IPv4 addresses with
additional text after the address and only quad-dotted notation without
trailing whitespaces. Some inet_aton() implementations ignore whitespace
and all data after whitespace, e.g. ‘127.0.0.1 whatever’.
- bpo-35907: CVE-2019-9948: Avoid file reading by disallowing local-file://
and local_file:// URL schemes in URLopener().open() and
URLopener().retrieve() of urllib.request.
- bpo-30458: Address CVE-2019-9740 by disallowing URL paths with embedded
whitespace or control characters through into the underlying http client
request. Such potentially malicious header injection URLs now cause an
http.client.InvalidURL exception to be raised.
- bpo-33529: Prevent fold function used in email header encoding from
entering infinite loop when there are too many non-ASCII characters in a
header.
- bpo-35755: shutil.which() now uses os.confstr("CS_PATH") if available and
if the PATH environment variable is not set. Remove also the current
directory from posixpath.defpath. On Unix, shutil.which() and the
subprocess module no longer search the executable in the current directory
if the PATH environment variable is not set.
Also remove the following upstreamed patches:
- 0033-bpo-36742-Fixes-handling-of-pre-normalization-charac.patch
- 0034-bpo-36742-Corrects-fix-to-handle-decomposition-in-us.patch
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[Peter: mention security fixes]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Globally change Adam Duskett's email address to aduskett@gmail.com.
Note that one or two of the patches may have been applied upstream with
the old email address, but in that case those patches will anyway be
removed when bumping.
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fixes CVE-2019-10160: urlsplit does not handle NFKC normalization (2nd fix)
While the fix for CVE-2019-9936 is included in 3.7.3, the followup
regression fixes unfortunatly aren't.
https://bugs.python.org/issue36742
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
