kumquat-buildroot/package/python3/0012-Add-importlib-fix-for-PEP-3147-issue.patch

100 lines
4.5 KiB
Diff
Raw Normal View History

From 9db508467be41bc35f06c0076c855b99a1519d24 Mon Sep 17 00:00:00 2001
From: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
Date: Wed, 22 Feb 2017 16:48:49 -0800
Subject: [PATCH] Add importlib fix for PEP 3147 issue
Python 3 has a new standard for installing .pyc file, called PEP
3147. Unfortunately, this standard requires both the .py and .pyc
files to be installed for a Python module to be found. This is quite
annoying on space-constrained embedded systems, since the .py file is
technically not required for execution.
This patch changes cache_from_source() and source_from_cache() in
importlib to get rid of the "__pycache__" directory.
This effectively disables PEP 3147 for:
* The python standard library
* Packages built with distutils or setuptools
* Packages built with automake that use the `py-compile` helper
Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
[ Andrey Smirnov: ported to Python 3.6 ]
Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
Lib/importlib/_bootstrap_external.py | 38 +++++-------------------------------
1 file changed, 5 insertions(+), 33 deletions(-)
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index 9feec50842..3550013d58 100644
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
python3: bump to 3.7.0 Other changes include: - Refreshing all necessary patches for 3.7.0 - Add a hash for the license file. - Python no longer has it's own internal libffi, as such, host-libffi is now required to build host-python3, and is added as a dependency. - Drop PYTHON3_LIBTOOL_PATCH = NO, since there is no longer any internal libffi copy that was causing the libtool patching process to fail. - A new core module "uuid" is now is added in the Config.in file, and relies on util-linux's uuid library. - Also, a new patch: 0030-Fix-cross-compiling-the-uuid-module.patch is required to fix compiling the uuid module, because the include directory search path for uuid.h is hardcoded to /usr/include/uuid, which causes an "unsafe for cross-compilation" error during compiling if the host pc has uuid headers installed. - 0031-Add-an-option-to-disable-uuid-module.patch is added to allow disabling the Python3 UUID module, so that when BR2_PACKAGE_PYTHON3_UUID is disabled by the UUID library is present, the uuid Python module is not built, as expected. - 0032-fix-building-on-older-distributions.patch is added to change os.replace by os.rename in the update_file.py script to fix building on older Linux distributions that have older versions of python that don't include os.replace. os.rename acts in the same way as os.replace, but is cross-platform compatible. Because Buildroot is guaranteed to be built in a POSIX environment, it is safe to change replace to rename. Tested on CentOS6 and Fedora28, All test results passed: br-arm-full [1/6]: OK br-arm-cortex-a9-glibc [2/6]: OK br-arm-cortex-m4-full [3/6]: SKIPPED br-x86-64-musl [4/6]: OK br-arm-full-static [5/6]: SKIPPED armv5-ctng-linux-gnueabi [6/6]: OK 6 builds, 2 skipped, 0 build failed, 0 legal-info failed Signed-off-by: Adam Duskett <aduskett@gmail.com> [Thomas: - remove PYTHON3_LIBTOOL_PATCH = NO - improve the solution in patch 0030-Fix-cross-compiling-the-uuid-module - add patch 0031-Add-an-option-to-disable-uuid-module] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-07-20 18:10:21 +02:00
@@ -283,8 +283,6 @@ def cache_from_source(path, debug_override=None, *, optimization=None):
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
a True value is the same as setting 'optimization' to the empty string
while a False value is equivalent to setting 'optimization' to '1'.
- If sys.implementation.cache_tag is None then NotImplementedError is raised.
-
"""
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
if debug_override is not None:
_warnings.warn('the debug_override parameter is deprecated; use '
python3: bump to 3.7.0 Other changes include: - Refreshing all necessary patches for 3.7.0 - Add a hash for the license file. - Python no longer has it's own internal libffi, as such, host-libffi is now required to build host-python3, and is added as a dependency. - Drop PYTHON3_LIBTOOL_PATCH = NO, since there is no longer any internal libffi copy that was causing the libtool patching process to fail. - A new core module "uuid" is now is added in the Config.in file, and relies on util-linux's uuid library. - Also, a new patch: 0030-Fix-cross-compiling-the-uuid-module.patch is required to fix compiling the uuid module, because the include directory search path for uuid.h is hardcoded to /usr/include/uuid, which causes an "unsafe for cross-compilation" error during compiling if the host pc has uuid headers installed. - 0031-Add-an-option-to-disable-uuid-module.patch is added to allow disabling the Python3 UUID module, so that when BR2_PACKAGE_PYTHON3_UUID is disabled by the UUID library is present, the uuid Python module is not built, as expected. - 0032-fix-building-on-older-distributions.patch is added to change os.replace by os.rename in the update_file.py script to fix building on older Linux distributions that have older versions of python that don't include os.replace. os.rename acts in the same way as os.replace, but is cross-platform compatible. Because Buildroot is guaranteed to be built in a POSIX environment, it is safe to change replace to rename. Tested on CentOS6 and Fedora28, All test results passed: br-arm-full [1/6]: OK br-arm-cortex-a9-glibc [2/6]: OK br-arm-cortex-m4-full [3/6]: SKIPPED br-x86-64-musl [4/6]: OK br-arm-full-static [5/6]: SKIPPED armv5-ctng-linux-gnueabi [6/6]: OK 6 builds, 2 skipped, 0 build failed, 0 legal-info failed Signed-off-by: Adam Duskett <aduskett@gmail.com> [Thomas: - remove PYTHON3_LIBTOOL_PATCH = NO - improve the solution in patch 0030-Fix-cross-compiling-the-uuid-module - add patch 0031-Add-an-option-to-disable-uuid-module] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-07-20 18:10:21 +02:00
@@ -296,10 +294,7 @@ def cache_from_source(path, debug_override=None, *, optimization=None):
path = _os.fspath(path)
head, tail = _path_split(path)
base, sep, rest = tail.rpartition('.')
- tag = sys.implementation.cache_tag
- if tag is None:
- raise NotImplementedError('sys.implementation.cache_tag is None')
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
- almost_filename = ''.join([(base if base else rest), sep, tag])
+ almost_filename = ''.join([(base if base else rest)])
if optimization is None:
if sys.flags.optimize == 0:
optimization = ''
python3: bump to 3.7.0 Other changes include: - Refreshing all necessary patches for 3.7.0 - Add a hash for the license file. - Python no longer has it's own internal libffi, as such, host-libffi is now required to build host-python3, and is added as a dependency. - Drop PYTHON3_LIBTOOL_PATCH = NO, since there is no longer any internal libffi copy that was causing the libtool patching process to fail. - A new core module "uuid" is now is added in the Config.in file, and relies on util-linux's uuid library. - Also, a new patch: 0030-Fix-cross-compiling-the-uuid-module.patch is required to fix compiling the uuid module, because the include directory search path for uuid.h is hardcoded to /usr/include/uuid, which causes an "unsafe for cross-compilation" error during compiling if the host pc has uuid headers installed. - 0031-Add-an-option-to-disable-uuid-module.patch is added to allow disabling the Python3 UUID module, so that when BR2_PACKAGE_PYTHON3_UUID is disabled by the UUID library is present, the uuid Python module is not built, as expected. - 0032-fix-building-on-older-distributions.patch is added to change os.replace by os.rename in the update_file.py script to fix building on older Linux distributions that have older versions of python that don't include os.replace. os.rename acts in the same way as os.replace, but is cross-platform compatible. Because Buildroot is guaranteed to be built in a POSIX environment, it is safe to change replace to rename. Tested on CentOS6 and Fedora28, All test results passed: br-arm-full [1/6]: OK br-arm-cortex-a9-glibc [2/6]: OK br-arm-cortex-m4-full [3/6]: SKIPPED br-x86-64-musl [4/6]: OK br-arm-full-static [5/6]: SKIPPED armv5-ctng-linux-gnueabi [6/6]: OK 6 builds, 2 skipped, 0 build failed, 0 legal-info failed Signed-off-by: Adam Duskett <aduskett@gmail.com> [Thomas: - remove PYTHON3_LIBTOOL_PATCH = NO - improve the solution in patch 0030-Fix-cross-compiling-the-uuid-module - add patch 0031-Add-an-option-to-disable-uuid-module] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-07-20 18:10:21 +02:00
@@ -310,40 +305,17 @@ def cache_from_source(path, debug_override=None, *, optimization=None):
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
if not optimization.isalnum():
raise ValueError('{!r} is not alphanumeric'.format(optimization))
almost_filename = '{}.{}{}'.format(almost_filename, _OPT, optimization)
- return _path_join(head, _PYCACHE, almost_filename + BYTECODE_SUFFIXES[0])
+ return _path_join(head, almost_filename + BYTECODE_SUFFIXES[0])
def source_from_cache(path):
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
"""Given the path to a .pyc. file, return the path to its .py file.
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
The .pyc file does not need to exist; this simply returns the path to
- the .py file calculated to correspond to the .pyc file. If path does
- not conform to PEP 3147/488 format, ValueError will be raised. If
- sys.implementation.cache_tag is None then NotImplementedError is raised.
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
-
+ the .py file calculated to correspond to the .pyc file.
"""
- if sys.implementation.cache_tag is None:
- raise NotImplementedError('sys.implementation.cache_tag is None')
- path = _os.fspath(path)
- head, pycache_filename = _path_split(path)
- head, pycache = _path_split(head)
- if pycache != _PYCACHE:
- raise ValueError('{} not bottom-level directory in '
- '{!r}'.format(_PYCACHE, path))
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
- dot_count = pycache_filename.count('.')
- if dot_count not in {2, 3}:
- raise ValueError('expected only 2 or 3 dots in '
- '{!r}'.format(pycache_filename))
python3: bump to 3.5.1 The major changes in terms of Buildroot packaging are: - Due to PEP488, Python no longer generates .pyc (unoptimized) and .pyo (optimized) byte-code files. Instead, it generates <foo>.pyc, <foo>.opt-1.pyc and <foo>.opt-2.pyc. Therefore, we removed the --disable-pyo-build option and kept only the --disable-pyc-build option, which completely disables building all .pyc files. In addition, since the optimized .opt-X.pyc files don't work if the corresponding un-optimized .pyc file is not present, we are for the moment unconditionally removing the optimized ones (keeping both the unoptimized and optimized ones doubles the required filesystem size!). So basically we preserve the behavior we had before this commit: BR2_PACKAGE_PYTHON3_PY_ONLY -> only *.py BR2_PACKAGE_PYTHON3_PYC_ONLY -> only non-optimized *.pyc BR2_PACKAGE_PYTHON3_PY_PYC -> both the *.py and non-optimized *.pyc To achieve this, the TARGET_FINALIZE_HOOKS are reworked: PYTHON3_REMOVE_PY_FILES is responsible for removing *.py files in the BR2_PACKAGE_PYTHON3_PYC_ONLY case. PYTHON3_REMOVE_PYC_FILES is responsible for removing *.pyc files in the BR2_PACKAGE_PYTHON3_PY_ONLY case. PYTHON3_REMOVE_OPTIMIZED_PYC_FILES is responsible for removing the optimized *.opt-1.pyc and *.opt-2.pyc files, which is done unconditionally. - The PEP3147 disabling patch had to be significantly reworked due to the code having changed heavily. The code was moved into a _bootstrap_external.py, which is a "frozen" Python module, i.e a module generated into a .h file at compile time using the _freeze_importlib program. - Due to the above, we now need to regenerate importlib.h at build time. Unfortunately, for the target Python _freeze_importlib is built for the target, so we can't run it on the build machine. To fix this, we copy the _freeze_importlib program from the host-python in $(HOST_DIR), and then patch the target python to use it. Since the same solution can be used for 'pgen', we do it, and avoid having to touch the graminit.{c,h} files. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-01 22:15:12 +02:00
- elif dot_count == 3:
- optimization = pycache_filename.rsplit('.', 2)[-2]
- if not optimization.startswith(_OPT):
- raise ValueError("optimization portion of filename does not start "
- "with {!r}".format(_OPT))
- opt_level = optimization[len(_OPT):]
- if not opt_level.isalnum():
- raise ValueError("optimization level {!r} is not an alphanumeric "
- "value".format(optimization))
- base_filename = pycache_filename.partition('.')[0]
+ head, filename = _path_split(path)
+ base_filename = filename.partition('.')[0]
return _path_join(head, base_filename + SOURCE_SUFFIXES[0])
--
2.13.5