kumquat-buildroot/docs/manual/adding-packages-python.txt

258 lines
10 KiB
Plaintext
Raw Normal View History

// -*- mode:doc; -*-
// vim: set syntax=asciidoc:
manual: use one-line titles instead of two-line titles (trivial) Asciidoc supports two syntaxes for section titles: two-line titles (title plus underline consisting of a particular symbol), and one-line titles (title prefixed with a specific number of = signs). The two-line title underlines are: Level 0 (top level): ====================== Level 1: ---------------------- Level 2: ~~~~~~~~~~~~~~~~~~~~~~ Level 3: ^^^^^^^^^^^^^^^^^^^^^^ Level 4 (bottom level): ++++++++++++++++++++++ and the one-line title prefixes: = Document Title (level 0) = == Section title (level 1) == === Section title (level 2) === ==== Section title (level 3) ==== ===== Section title (level 4) ===== The buildroot manual is currenly using the two-line titles, but this has multiple disadvantages: - asciidoc also uses some of the underline symbols for other purposes (like preformatted code, example blocks, ...), which makes it difficult to do mass replacements, such as a planned follow-up patch that needs to move all sections one level down. - it is difficult to remember which level a given underline symbol (=-~^+) corresponds to, while counting = signs is easy. This patch changes all two-level titles to one-level titles in the manual. The bulk of the change was done with the following Python script, except for the level 1 titles (-----) as these underlines are also used for literal code blocks. This patch only changes the titles, no other changes. In adding-packages-directory.txt, I did add missing newlines between some titles and their content. ---------------------------------------------------------------------------- #!/usr/bin/env python import sys import mmap import re for input in sys.argv[1:]: f = open(input, 'r+') f.flush() s = mmap.mmap(f.fileno(), 0) # Level 0 (top level): ====================== = # Level 1: ---------------------- == # Level 2: ~~~~~~~~~~~~~~~~~~~~~~ === # Level 3: ^^^^^^^^^^^^^^^^^^^^^^ ==== # Level 4 (bottom level): ++++++++++++++++++++++ ===== def replace_title(s, symbol, replacement): pattern = re.compile(r'(.+\n)\%s{2,}\n' % symbol, re.MULTILINE) return pattern.sub(r'%s \1' % replacement, s) new = s new = replace_title(new, '=', '=') new = replace_title(new, '+', '=====') new = replace_title(new, '^', '====') new = replace_title(new, '~', '===') #new = replace_title(new, '-', '==') s.seek(0) s.write(new) s.resize(s.tell()) s.close() f.close() ---------------------------------------------------------------------------- Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-05-02 07:47:30 +02:00
=== Infrastructure for Python packages
This infrastructure applies to Python packages that use the standard
Python setuptools mechanism as their build system, generally
recognizable by the usage of a +setup.py+ script.
[[python-package-tutorial]]
manual: use one-line titles instead of two-line titles (trivial) Asciidoc supports two syntaxes for section titles: two-line titles (title plus underline consisting of a particular symbol), and one-line titles (title prefixed with a specific number of = signs). The two-line title underlines are: Level 0 (top level): ====================== Level 1: ---------------------- Level 2: ~~~~~~~~~~~~~~~~~~~~~~ Level 3: ^^^^^^^^^^^^^^^^^^^^^^ Level 4 (bottom level): ++++++++++++++++++++++ and the one-line title prefixes: = Document Title (level 0) = == Section title (level 1) == === Section title (level 2) === ==== Section title (level 3) ==== ===== Section title (level 4) ===== The buildroot manual is currenly using the two-line titles, but this has multiple disadvantages: - asciidoc also uses some of the underline symbols for other purposes (like preformatted code, example blocks, ...), which makes it difficult to do mass replacements, such as a planned follow-up patch that needs to move all sections one level down. - it is difficult to remember which level a given underline symbol (=-~^+) corresponds to, while counting = signs is easy. This patch changes all two-level titles to one-level titles in the manual. The bulk of the change was done with the following Python script, except for the level 1 titles (-----) as these underlines are also used for literal code blocks. This patch only changes the titles, no other changes. In adding-packages-directory.txt, I did add missing newlines between some titles and their content. ---------------------------------------------------------------------------- #!/usr/bin/env python import sys import mmap import re for input in sys.argv[1:]: f = open(input, 'r+') f.flush() s = mmap.mmap(f.fileno(), 0) # Level 0 (top level): ====================== = # Level 1: ---------------------- == # Level 2: ~~~~~~~~~~~~~~~~~~~~~~ === # Level 3: ^^^^^^^^^^^^^^^^^^^^^^ ==== # Level 4 (bottom level): ++++++++++++++++++++++ ===== def replace_title(s, symbol, replacement): pattern = re.compile(r'(.+\n)\%s{2,}\n' % symbol, re.MULTILINE) return pattern.sub(r'%s \1' % replacement, s) new = s new = replace_title(new, '=', '=') new = replace_title(new, '+', '=====') new = replace_title(new, '^', '====') new = replace_title(new, '~', '===') #new = replace_title(new, '-', '==') s.seek(0) s.write(new) s.resize(s.tell()) s.close() f.close() ---------------------------------------------------------------------------- Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-05-02 07:47:30 +02:00
==== +python-package+ tutorial
First, let's see how to write a +.mk+ file for a Python package,
with an example :
------------------------
01: ################################################################################
02: #
03: # python-foo
04: #
05: ################################################################################
06:
07: PYTHON_FOO_VERSION = 1.0
08: PYTHON_FOO_SOURCE = python-foo-$(PYTHON_FOO_VERSION).tar.xz
09: PYTHON_FOO_SITE = http://www.foosoftware.org/download
10: PYTHON_FOO_LICENSE = BSD-3-Clause
11: PYTHON_FOO_LICENSE_FILES = LICENSE
12: PYTHON_FOO_ENV = SOME_VAR=1
13: PYTHON_FOO_DEPENDENCIES = libmad
14: PYTHON_FOO_SETUP_TYPE = distutils
15:
16: $(eval $(python-package))
------------------------
On line 7, we declare the version of the package.
On line 8 and 9, we declare the name of the tarball (xz-ed tarball
recommended) and the location of the tarball on the Web. Buildroot
will automatically download the tarball from this location.
On line 10 and 11, we give licensing details about the package (its
license on line 10, and the file containing the license text on line
11).
On line 12, we tell Buildroot to pass custom options to the Python
+setup.py+ script when it is configuring the package.
On line 13, we declare our dependencies, so that they are built
before the build process of our package starts.
On line 14, we declare the specific Python build system being used. In
this case the +distutils+ Python build system is used. The two
supported ones are +distutils+ and +setuptools+.
Finally, on line 16, we invoke the +python-package+ macro that
generates all the Makefile rules that actually allow the package to be
built.
[[python-package-reference]]
manual: use one-line titles instead of two-line titles (trivial) Asciidoc supports two syntaxes for section titles: two-line titles (title plus underline consisting of a particular symbol), and one-line titles (title prefixed with a specific number of = signs). The two-line title underlines are: Level 0 (top level): ====================== Level 1: ---------------------- Level 2: ~~~~~~~~~~~~~~~~~~~~~~ Level 3: ^^^^^^^^^^^^^^^^^^^^^^ Level 4 (bottom level): ++++++++++++++++++++++ and the one-line title prefixes: = Document Title (level 0) = == Section title (level 1) == === Section title (level 2) === ==== Section title (level 3) ==== ===== Section title (level 4) ===== The buildroot manual is currenly using the two-line titles, but this has multiple disadvantages: - asciidoc also uses some of the underline symbols for other purposes (like preformatted code, example blocks, ...), which makes it difficult to do mass replacements, such as a planned follow-up patch that needs to move all sections one level down. - it is difficult to remember which level a given underline symbol (=-~^+) corresponds to, while counting = signs is easy. This patch changes all two-level titles to one-level titles in the manual. The bulk of the change was done with the following Python script, except for the level 1 titles (-----) as these underlines are also used for literal code blocks. This patch only changes the titles, no other changes. In adding-packages-directory.txt, I did add missing newlines between some titles and their content. ---------------------------------------------------------------------------- #!/usr/bin/env python import sys import mmap import re for input in sys.argv[1:]: f = open(input, 'r+') f.flush() s = mmap.mmap(f.fileno(), 0) # Level 0 (top level): ====================== = # Level 1: ---------------------- == # Level 2: ~~~~~~~~~~~~~~~~~~~~~~ === # Level 3: ^^^^^^^^^^^^^^^^^^^^^^ ==== # Level 4 (bottom level): ++++++++++++++++++++++ ===== def replace_title(s, symbol, replacement): pattern = re.compile(r'(.+\n)\%s{2,}\n' % symbol, re.MULTILINE) return pattern.sub(r'%s \1' % replacement, s) new = s new = replace_title(new, '=', '=') new = replace_title(new, '+', '=====') new = replace_title(new, '^', '====') new = replace_title(new, '~', '===') #new = replace_title(new, '-', '==') s.seek(0) s.write(new) s.resize(s.tell()) s.close() f.close() ---------------------------------------------------------------------------- Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-05-02 07:47:30 +02:00
==== +python-package+ reference
As a policy, packages that merely provide Python modules should all be
named +python-<something>+ in Buildroot. Other packages that use the
Python build system, but are not Python modules, can freely choose
their name (existing examples in Buildroot are +scons+ and
+supervisor+).
In their +Config.in+ file, they should depend on +BR2_PACKAGE_PYTHON+
so that when Buildroot will enable Python 3 usage for modules, we will
be able to enable Python modules progressively on Python 3.
The main macro of the Python package infrastructure is
+python-package+. It is similar to the +generic-package+ macro. It is
also possible to create Python host packages with the
+host-python-package+ macro.
Just like the generic infrastructure, the Python infrastructure works
by defining a number of variables before calling the +python-package+
or +host-python-package+ macros.
All the package metadata information variables that exist in the
xref:generic-package-reference[generic package infrastructure] also
exist in the Python infrastructure: +PYTHON_FOO_VERSION+,
+PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+, +PYTHON_FOO_SITE+,
+PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+, +PYTHON_FOO_LICENSE+,
+PYTHON_FOO_LICENSE_FILES+, +PYTHON_FOO_INSTALL_STAGING+, etc.
Note that:
* It is not necessary to add +python+ or +host-python+ in the
+PYTHON_FOO_DEPENDENCIES+ variable of a package, since these basic
dependencies are automatically added as needed by the Python
package infrastructure.
* Similarly, it is not needed to add +host-setuptools+ and/or
+host-distutilscross+ dependencies to +PYTHON_FOO_DEPENDENCIES+ for
setuptools-based packages, since these are automatically added by
the Python infrastructure as needed.
One variable specific to the Python infrastructure is mandatory:
* +PYTHON_FOO_SETUP_TYPE+, to define which Python build system is used
by the package. The two supported values are +distutils+ and
+setuptools+. If you don't know which one is used in your package,
look at the +setup.py+ file in your package source code, and see
whether it imports things from the +distutils+ module or the
+setuptools+ module.
A few additional variables, specific to the Python infrastructure, can
optionally be defined, depending on the package's needs. Many of them
are only useful in very specific cases, typical packages will
therefore only use a few of them, or none.
* +PYTHON_FOO_SUBDIR+ may contain the name of a subdirectory inside the
package that contains the main +setup.py+ file. This is useful,
if for example, the main +setup.py+ file is not at the root of
the tree extracted by the tarball. If +HOST_PYTHON_FOO_SUBDIR+ is not
specified, it defaults to +PYTHON_FOO_SUBDIR+.
* +PYTHON_FOO_ENV+, to specify additional environment variables to
pass to the Python +setup.py+ script (for both the build and install
steps). Note that the infrastructure is automatically passing
several standard variables, defined in +PKG_PYTHON_DISTUTILS_ENV+
(for distutils target packages), +HOST_PKG_PYTHON_DISTUTILS_ENV+
(for distutils host packages), +PKG_PYTHON_SETUPTOOLS_ENV+ (for
setuptools target packages) and +HOST_PKG_PYTHON_SETUPTOOLS_ENV+
(for setuptools host packages).
* +PYTHON_FOO_BUILD_OPTS+, to specify additional options to pass to the
Python +setup.py+ script during the build step. For target distutils
packages, the +PKG_PYTHON_DISTUTILS_BUILD_OPTS+ options are already
passed automatically by the infrastructure.
* +PYTHON_FOO_INSTALL_TARGET_OPTS+, +PYTHON_FOO_INSTALL_STAGING_OPTS+,
+HOST_PYTHON_FOO_INSTALL_OPTS+ to specify additional options to pass
to the Python +setup.py+ script during the target installation step,
the staging installation step or the host installation,
respectively. Note that the infrastructure is automatically passing
some options, defined in +PKG_PYTHON_DISTUTILS_INSTALL_TARGET_OPTS+
or +PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS+ (for target distutils
packages), +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS+ (for host
distutils packages), +PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS+ or
+PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS+ (for target setuptools
packages) and +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS+ (for host
setuptools packages).
pkg-python: support host-python dependency different from the python in the target Some packages need a host-python interpreter with a version different from the one installed in the target to run some build scripts (eg. scons requires python2 to run, to build any kind of packages even if the python interpreter selected for the target is python3). In such cases, we need to add the right host-python dependency to the package using the host-python-package infrastructure, and we also want to invoke the right host python interpreter during the build steps. This patch adds a *_NEEDS_HOST_PYTHON variable that can be set either to 'python2' or 'python3'. This variable can be set by any package using the host-python-package infrastructure to force the python interpreter for the build. This variable also takes care of setting the right host-python dependency. This *_NEEDS_HOST_PYTHON variable only affects packages using the host-python-package infrastructure. If some configure/build/install commands are overloaded in the *.mk file, the right python interpreter should be explicitly called. If the package defines some tool variable (eg.: SCONS), the variable should explicitly call the right python interpreter. [Thomas: - fixes to the commit log and documentation suggested by Yann - rename the variable from <pkg>_FORCE_HOST_PYTHON to <pkg>_NEEDS_HOST_PYTHON, as suggested by Yann - do not allow any other value than python2 and python3 in <pkg>_NEEDS_HOST_PYTHON, as suggested by Yann.] Signed-off-by: Samuel Martin <s.martin49@gmail.com> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2014-03-05 23:04:42 +01:00
* +HOST_PYTHON_FOO_NEEDS_HOST_PYTHON+, to define the host python
interpreter. The usage of this variable is limited to host
packages. The two supported value are +python2+ and +python3+. It
will ensure the right host python package is available and will
pkg-python: support host-python dependency different from the python in the target Some packages need a host-python interpreter with a version different from the one installed in the target to run some build scripts (eg. scons requires python2 to run, to build any kind of packages even if the python interpreter selected for the target is python3). In such cases, we need to add the right host-python dependency to the package using the host-python-package infrastructure, and we also want to invoke the right host python interpreter during the build steps. This patch adds a *_NEEDS_HOST_PYTHON variable that can be set either to 'python2' or 'python3'. This variable can be set by any package using the host-python-package infrastructure to force the python interpreter for the build. This variable also takes care of setting the right host-python dependency. This *_NEEDS_HOST_PYTHON variable only affects packages using the host-python-package infrastructure. If some configure/build/install commands are overloaded in the *.mk file, the right python interpreter should be explicitly called. If the package defines some tool variable (eg.: SCONS), the variable should explicitly call the right python interpreter. [Thomas: - fixes to the commit log and documentation suggested by Yann - rename the variable from <pkg>_FORCE_HOST_PYTHON to <pkg>_NEEDS_HOST_PYTHON, as suggested by Yann - do not allow any other value than python2 and python3 in <pkg>_NEEDS_HOST_PYTHON, as suggested by Yann.] Signed-off-by: Samuel Martin <s.martin49@gmail.com> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2014-03-05 23:04:42 +01:00
invoke it for the build. If some build steps are overloaded, the
right python interpreter must be explicitly called in the commands.
With the Python infrastructure, all the steps required to build and
install the packages are already defined, and they generally work well
for most Python-based packages. However, when required, it is still
possible to customize what is done in any particular step:
* By adding a post-operation hook (after extract, patch, configure,
build or install). See xref:hooks[] for details.
* By overriding one of the steps. For example, even if the Python
infrastructure is used, if the package +.mk+ file defines its own
+PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the
default Python one. However, using this method should be restricted
to very specific cases. Do not use it in the general case.
[[scanpypi]]
==== Generating a +python-package+ from a PyPI repository
If the Python package for which you would like to create a Buildroot
package is available on PyPI, you may want to use the +scanpypi+ tool
located in +utils/+ to automate the process.
You can find the list of existing PyPI packages
https://pypi.python.org[here].
+scanpypi+ requires Python's +setuptools+ package to be installed on
your host.
When at the root of your buildroot directory just do :
-----------------------
utils/scanpypi foo bar -o package
-----------------------
This will generate packages +python-foo+ and +python-bar+ in the package
folder if they exist on https://pypi.python.org.
Find the +external python modules+ menu and insert your package inside.
Keep in mind that the items inside a menu should be in alphabetical order.
Please keep in mind that you'll most likely have to manually check the
package for any mistakes as there are things that cannot be guessed by
the generator (e.g. dependencies on any of the python core modules
such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the
license and license files are guessed and must be checked. You also
need to manually add the package to the +package/Config.in+ file.
If your Buildroot package is not in the official Buildroot tree but in
a br2-external tree, use the -o flag as follows:
-----------------------
utils/scanpypi foo bar -o other_package_dir
-----------------------
This will generate packages +python-foo+ and +python-bar+ in the
+other_package_directory+ instead of +package+.
Option +-h+ will list the available options:
-----------------------
utils/scanpypi -h
-----------------------
[[python-package-cffi-backend]]
==== +python-package+ CFFI backend
C Foreign Function Interface for Python (CFFI) provides a convenient
and reliable way to call compiled C code from Python using interface
declarations written in C. Python packages relying on this backend can
be identified by the appearance of a +cffi+ dependency in the
+install_requires+ field of their +setup.py+ file.
Such a package should:
* add +python-cffi+ as a runtime dependency in order to install the
compiled C library wrapper on the target. This is achieved by adding
+select BR2_PACKAGE_PYTHON_CFFI+ to the package +Config.in+.
------------------------
config BR2_PACKAGE_PYTHON_FOO
bool "python-foo"
select BR2_PACKAGE_PYTHON_CFFI # runtime
------------------------
* add +host-python-cffi+ as a build-time dependency in order to
cross-compile the C wrapper. This is achieved by adding
+host-python-cffi+ to the +PYTHON_FOO_DEPENDENCIES+ variable.
------------------------
################################################################################
#
# python-foo
#
################################################################################
...
PYTHON_FOO_DEPENDENCIES = host-python-cffi
$(eval $(python-package))
------------------------