kumquat-buildroot/support/scripts/pycompile.py

94 lines
2.9 KiB
Python
Raw Normal View History

python/python3: globalize *.pyc files compilation Currently, each python package (be it the python interpreter package itself or external python modules) is responsible for compiling its .py into .pyc files. Unfortunately, this is not ideal as some packages only install .py files without compiling them into .pyc files. In this case, if the Buildroot configuration specifies to keep only the .pyc files, the .py files are removed and lost. To address this, this commit changes the logic by making the compilation of .pyc files a global operation: the python interpreter packages register a target finalize hook that is in charge of compiling all installed .py files. The *.pyc generation on a per package basis is disabled in the python-package infrastructure by passing the "--no-compile" option to setup.py. The *.pyc generation for the Python interpreter internal modules is disabled through --disable-pyc-build configure option. A small helper script is used to perform the compilation, the purpose of this script is to abort the compilation process if one of the .py file cannot be compiled. It has been provided by Samuel Martin and integrated into this commit. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Cc: Samuel Martin <s.martin49@gmail.com> [Thomas: - rework for python 3.5 - integrate Samuel proposal that allows to detect compilation failures.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-17 23:19:15 +02:00
#!/usr/bin/env python
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
"""
Byte compile all .py files from provided directories. This script is an
alternative implementation of compileall.compile_dir written with
cross-compilation in mind.
"""
python/python3: globalize *.pyc files compilation Currently, each python package (be it the python interpreter package itself or external python modules) is responsible for compiling its .py into .pyc files. Unfortunately, this is not ideal as some packages only install .py files without compiling them into .pyc files. In this case, if the Buildroot configuration specifies to keep only the .pyc files, the .py files are removed and lost. To address this, this commit changes the logic by making the compilation of .pyc files a global operation: the python interpreter packages register a target finalize hook that is in charge of compiling all installed .py files. The *.pyc generation on a per package basis is disabled in the python-package infrastructure by passing the "--no-compile" option to setup.py. The *.pyc generation for the Python interpreter internal modules is disabled through --disable-pyc-build configure option. A small helper script is used to perform the compilation, the purpose of this script is to abort the compilation process if one of the .py file cannot be compiled. It has been provided by Samuel Martin and integrated into this commit. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Cc: Samuel Martin <s.martin49@gmail.com> [Thomas: - rework for python 3.5 - integrate Samuel proposal that allows to detect compilation failures.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-17 23:19:15 +02:00
from __future__ import print_function
import argparse
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
import os
import py_compile
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
import re
import sys
python/python3: globalize *.pyc files compilation Currently, each python package (be it the python interpreter package itself or external python modules) is responsible for compiling its .py into .pyc files. Unfortunately, this is not ideal as some packages only install .py files without compiling them into .pyc files. In this case, if the Buildroot configuration specifies to keep only the .pyc files, the .py files are removed and lost. To address this, this commit changes the logic by making the compilation of .pyc files a global operation: the python interpreter packages register a target finalize hook that is in charge of compiling all installed .py files. The *.pyc generation on a per package basis is disabled in the python-package infrastructure by passing the "--no-compile" option to setup.py. The *.pyc generation for the Python interpreter internal modules is disabled through --disable-pyc-build configure option. A small helper script is used to perform the compilation, the purpose of this script is to abort the compilation process if one of the .py file cannot be compiled. It has been provided by Samuel Martin and integrated into this commit. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Cc: Samuel Martin <s.martin49@gmail.com> [Thomas: - rework for python 3.5 - integrate Samuel proposal that allows to detect compilation failures.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2016-05-17 23:19:15 +02:00
def compile_one(host_path, strip_root=None, verbose=False):
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
"""
Compile a .py file into a .pyc file located next to it.
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
:arg host_path:
Absolute path to the file to compile on the host running the build.
:arg strip_root:
Prefix to remove from the original source paths encoded in compiled
files.
:arg verbose:
Print compiled file paths.
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
"""
if os.path.islink(host_path) or not os.path.isfile(host_path):
return # only compile real files
if not re.match(r"^[_A-Za-z][_A-Za-z0-9]*\.py$",
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
os.path.basename(host_path)):
return # only compile "importable" python modules
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
if strip_root is not None:
# determine the runtime path of the file (i.e.: relative path to root
# dir prepended with "/").
runtime_path = os.path.join("/", os.path.relpath(host_path, strip_root))
else:
runtime_path = host_path
if verbose:
print(" PYC {}".format(runtime_path))
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
# will raise an error if the file cannot be compiled
py_compile.compile(host_path, cfile=host_path + "c",
dfile=runtime_path, doraise=True)
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
def existing_dir_abs(arg):
"""
argparse type callback that checks that argument is a directory and returns
its absolute path.
"""
if not os.path.isdir(arg):
raise argparse.ArgumentTypeError('no such directory: {!r}'.format(arg))
return os.path.abspath(arg)
def main():
parser = argparse.ArgumentParser(description=__doc__)
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
parser.add_argument("dirs", metavar="DIR", nargs="+", type=existing_dir_abs,
help="Directory to recursively scan and compile")
parser.add_argument("--strip-root", metavar="ROOT", type=existing_dir_abs,
help="""
Prefix to remove from the original source paths encoded
in compiled files
""")
parser.add_argument("--verbose", action="store_true",
help="Print compiled files")
args = parser.parse_args()
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
try:
for d in args.dirs:
if args.strip_root and ".." in os.path.relpath(d, args.strip_root):
parser.error("DIR: not inside ROOT dir: {!r}".format(d))
for parent, _, files in os.walk(d):
for f in files:
compile_one(os.path.join(parent, f), args.strip_root,
args.verbose)
support/scripts/pycompile: fix .pyc original source file paths When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() directly with pertinent options: - The script now has a new --strip-root argument. This argument is optional but will always be specified when compiling py files in buildroot. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using --strip-root=$(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Previously, we had a --force option to tell compileall.compiledir() to forcibly recompile files if they had changed. Now, we would have to handle it ourselves. It turns out to not be easy and would need us to delve into the format of bytecompiled files to extract metadata and compare it with the expected values, that being even dependent on the python version being used (fortunately, only two for us: python 2.7 and the latext 3.x). Still, this is deemed too complex, and byte-compiling is pretty fast, so much so that it should be eclipsed by the build duration anyway. So we just drop support for --force, and instead we always byte-compile. Signed-off-by: Julien Floret <julien.floret@6wind.com> Signed-off-by: Robin Jarry <robin.jarry@6wind.com> [yann.morin.1998@free.fr: - always byte-compile - drop --force - expand commit log to state so and explain why ] Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-09-10 10:32:51 +02:00
except Exception as e:
print("error: {}".format(e))
return 1
return 0
if __name__ == "__main__":
sys.exit(main())