Project:Python/Python.eclass conversion

This guide provides tips and guidelines for converting python.eclass packages.

Generic tips

 * 1) Always bump revision when converting. This is a big enough change, and we'd like to make sure that upgrade wipes out all references to the old eclass.
 * 2) If in doubt, ask on #gentoo-python. We're there to help.
 * 3) Please remember to test the packages after conversion. If the test suite fails the same as before, it's ok, but if the conversion results in new tests failing then likely something went wrong.
 * 4) Compare the installed file list before and after conversion. Make sure nothing got lost :).
 * 5) Ensure that Python modules are byte-compiled (.pyc, .pyo). Many packages fail to do that currently.
 * 6) Many of python.eclass ebuilds are over-complex or even broken. Do not attempt to do 1:1 conversion. Instead, try to improve the ebuild.
 * 7) Not everything needs to be installed for all Python implementations. python-single-r1 is easier to use, so use it when multi-install isn't necessary. Value your time, we can re-add multiple implementation support later if necessary.

Eclass choice
The following table maps old eclasses to the new ones.

(*) If support for multiple implementations is complex and not really required by reverse dependencies, python-single-r1 can be used instead.

Ebuild head
All python.eclass ebuilds start with a few variables that are set before the inherit line. Those are:


 * 1) PYTHON_DEPEND to specify dependency atom on Python interpreter (using a micro-syntax).
 * 2) PYTHON_USE_WITH, PYTHON_USE_WITH_OR, PYTHON_USE_WITH_OPT to handle requiring USE flags on the Python interpreter.
 * 3) SUPPORT_PYTHON_ABIS to enable multi-impl support, and RESTRICT_PYTHON_ABIS to specify opt-out list of supported implementations.

All python-r1 suite eclasses replace those five variables with the following two:


 * 1) PYTHON_COMPAT to list supported Python implementations (opt-in, bash array).
 * 2) PYTHON_REQ_USE to specify USE dependency on the Python interpreter (plain USE dependency string).

When deciding which Python implementations to put in PYTHON_COMPAT, please prefer testing over trusting the current values. Additionally check the dependencies. If the dependencies can't support a particular Python implementation, your package must not claim to support it either (otherwise the cross-package dependency couldn't be fulfilled).

The value of RESTRICT_PYTHON_ABIS can be used as a tip; however note that opt-out nature of this variable often means that uncommon implementations such as PyPy or Jython were never tested. If it's not set, look at PYTHON_DEPEND. Remember to remove implementations that are not supported anymore (older than 2.7 and 3.3 respectively). For mapping of old eclass implementation names, look at implementations article.

PYTHON_DEPEND may contain a USE flag name as well. The mapping of this sub-syntax is covered in the ebuild metadata section.

The value of PYTHON_USE_WITH can be usually copied directly to PYTHON_REQ_USE. However, if some of the supported implementations do not support (require) the particular flag, (-) and (+) USE defaults should be appended respectively. A cheat sheet on recommended USE defaults is available in the Implementation USE flags article.

Complex values of PYTHON_USE_WITH as well as uses of PYTHON_USE_WITH_OR and PYTHON_USE_WITH_OPT are not supported.

Metadata variables
The RDEPEND and DEPEND variables are filled implicitly by the eclass based on the value of PYTHON_DEPEND variable. If the variable specifies version ranges only, the dependency is added as an unconditional dependency. If it is prefixed by a USE flag name and a question mark (e.g. ), an appropriate USE flag conditional is added.

In python-r1 suite of eclasses, the ebuild developer is responsible for filling in RDEPEND and DEPEND manually, as appropriate. The eclasses set PYTHON_DEPS helper variable which contains proper dependency atoms. Depending on the structure of Python dependency, the variable can be either referenced unconditionally in RDEPEND and DEPEND, or placed within a USE conditional block.

Please note that in most cases the Python interpreter dependency needs to be specified both in RDEPEND and in DEPEND. The pure runtime dependency occurs only when no Python references is done during build-time, e.g. when installing trivial scripts without byte-compiling or obtaining interpreter information. The pure build-time dependency is to be combined with python-any-r1 eclass.

The python-r1 and python-single-r1 eclasses add two additional requirements:


 * 1) All dependencies on other Python packages need to carry the value of PYTHON_USEDEP as a USE dependency.
 * 2) The value of PYTHON_REQUIRED_USE needs to be referenced in REQUIRED_USE, with USE conditionals matching the ones in RDEPEND and DEPEND.

pkg_setup
The python.eclass packages usually call the following functions during pkg_setup:


 * 1) python_set_active_version to enforce a specific Python version in global scope (alike PYTHON_DEPEND),
 * 2) python_pkg_setup to perform implementation correctness checks and initialize start implementation in multi-impl packages.

In python-r1 suite, the correctness checks are done implicitly through dependencies. The implementation choice is performed by python_setup function which uses the value of PYTHON_COMPAT. However, the function rarely needs to be called directly. Depending on the eclass used:


 * 1) python-single-r1 and python-any-r1 eclasses export pkg_setup which finds and enable correct Python interpreter globally, if source build is performed. If Python is needed during pkg_{pre,post}* functions when a binary package install is performed, python_setup needs to be called directly instead.
 * 2) python-r1 does not export pkg_setup. Instead, Python is initialized (locally for the called function) when python_foreach_impl is used. If Python is needed outside those calls, python_setup can be called directly to initialize the environment for best implementation selected.

Python byte-code compilation
The python.eclass enforced specific rules for Python byte-code compilation:


 * 1) packages shouldn't have compiled Python modules during build, unless needed for some reason (poorly written tests etc.),
 * 2) packages mustn't have installed compiled Python modules (.pyc, .pyo files) to the install root,
 * 3) packages should have compiled modules into byte-code in pkg_postinst, and removed stray byte-code in pkg_postrm.

The python-r1 suite requires quite the opposite: packages need to install byte-compiled modules to the install root.

Therefore, when converting ebuilds:


 * 1) python_mod_optimize and python_mod_cleanup calls have to be removed, often making pkg_postinst and pkg_postrm unnecessary.
 * 2) If the ebuild is applying patches or other hacks to disable byte-compilation (such as overwriting py-compile script), those have to be removed.
 * 3) If the package does not byte-compile installed files even after removing all the hacks, python_optimize function can be used to byte-compile modules.

Please make sure that byte-code is compiled and optimized correctly. This specifically means that:


 * 1) Python 2.* creates .pyc files in the same directory as .py, while Python 3.* uses a __pycache__ subdirectory.
 * 2) CPython creates both .pyc & .pyo files, PyPy creates only .pyc.

If .pyc files are not placed in a subdirectory with Python 3, py-compile script may need to be updated (from automake distribution). If .pyo files are not created, python_optimize may need to be used.

$(PYTHON), ${EPYTHON}
The python.eclass ebuilds often use various features of the all-in-one $(PYTHON) getter-function. Additionally, in some contexts ${EPYTHON} variable is used to reference the current implementation name.

The python-r1 suite provides two well-defined variables instead: PYTHON and EPYTHON, and appropriate contexts to use them.

python_convert_shebangs
The python_convert_shebangs function provided by python.eclass is used to change shebangs in Python scripts to a requested value. As a replacement, python-r1 suite provides python_fix_shebang function which updates shebangs to match the requested implementation.

It should be noted that the functions differ in usage. python_convert_shebangs takes a Python version followed by file or directory names. python_fix_shebang takes only path list, and uses the Python version currently in use (set by python_setup or python_foreach_impl). This matches one of the goals of python-r1 suite — always matching Python implementations explicitly and discouraging uses of generic python2 and python3 shebangs.

python_fix_shebang is usually used in python-single-r1 and python-any-r1 only. In multi-impl (python-r1) ebuilds, a few alternatives need to be considered:


 * 1) if a build system installs a Python script with generic shebang, python_replicate_script can be used to add the script for all supported implementations.
 * 2) If the script is installed manually, python_doscript can be used inside python_foreach_impl loop to install the script (instead of dobin).

In case of scripts that are used only during build-time, shebang substitution is unnecessary if they use #!/usr/bin/env python[23] shebangs. The eclass ensure that proper wrappers are put in PATH for those executables.

python_execute_function
The python_execute_function runs the specified function or command with setup for one or more Python implementations. Depending on the exact use, python-r1 provides two replacements:


 * 1) python_foreach_impl to execute the function for all enabled Python implementations (the normal behavior),
 * 2) Project:Python/python-r1 to setup the environment for the 'best' Python implementation, with the function being called (or inlined) afterwards (replacement for python_execute_function -f).

It should be noted that unlike in python.eclass, python_foreach_impl does not terminate the build if function returns non-zero value (|| return 1 affects only python_foreach_impl return status). If necessary, the function needs to call die explicitly.

PYTHON_CFLAGS
python.eclass provides four flag manipulation variables:


 * 1) PYTHON_CFLAGS,
 * 2) PYTHON_CPPFLAGS,
 * 3) PYTHON_CXXFLAGS,
 * 4) PYTHON_LDFLAGS.

All those variables support adding or removing flags (+ or - operator, respectively) for Python implementations matching a pattern. This feature is not supported by distutils-r1, and any flag manipulation need to be performed manually.

Divergences from python-r1
distutils-r1 is built on top of python-r1, and therefore most of the instructions for python.eclass apply there as well. However, there are a few notable exceptions:


 * 1) python-r1 or python-single-r1 is used as a backend, depending on whether DISTUTILS_SINGLE_IMPL is unset or set appropriately.
 * 2) RDEPEND, DEPEND and REQUIRED_USE are filled in automatically, unless DISTUTILS_OPTIONAL is set.

You can use the two fore-mentioned variables to modify the eclass behavior as necessary for the ported package.

Phase functions
Both distutils.eclass and distutils-r1 export a few phase functions by default. The following table compares exported functions.

It should be also noted that distutils.eclass provides opaque phase functions while distutils-r1 splits each phase function into two sub-phases. This has two important implications:


 * 1) Changing per-impl behavior with distutils.eclass required using python_execute_function and sometimes even inlining parts of the original phase function. In distutils-r1 you can override the sub-phase instead, and call the default implementation inside if necessary.
 * 2) Complete Python environment setup is available inside the sub-phase functions, reducing the number of necessary eclass function calls and variable declarations. In particular, PYTHONPATH is set properly to reference built Python packages.

PYTHON_MODNAME
The PYTHON_MODNAME specified list of packages installed by the package, and caused the respective packages to be compiled in pkg_postinst and cleansed in pkg_postrm. Since the python-r1 suite no longer does byte-compilation in pkg_postinst, and distutils byte-compiles installed Python modules, all references to PYTHON_MODNAME can be removed safely.

DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES
The DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES causes distutils.eclass to create a separate copy of package sources for each Python implementation, and spawn setup.py from those separate copies. The distutils-r1 equivalent variable is called DISTUTILS_IN_SOURCE_BUILD.

DISTUTILS_SETUP_FILES
The DISTUTILS_SETUP_FILES provided support for running setup.py files, possibly multiple, in non-standard paths or using non-standard names. In distutils-r1, those corner cases need to be handled manually.

If setup.py is located in directory other than the default value of ${S}, either S needs to be adjusted or the working directory changed before running distutils-r1_src_* (or distutils-r1_python_*). If multiple setup files need to be run, the fore-mentioned phase function may be called multiple times in different locations.

DISTUTILS_GLOBAL_OPTIONS
The DISTUTILS_GLOBAL_OPTIONS provided support for specifying additional parameters passed to setup.py before the commands. The distutils-r1 equivalent is mydistutilsargs, with the additional recommendation to set it in python_configure_all sub-phase.

DISTUTILS_GLOBAL_OPTIONS provided also support for specifying parameters per-implementation. This is not explicitly supported in distutils-r1. Instead, the ebuild needs to set mydistutilsargs locally as appropriate, e.g. within python_compile and python_install calls.

DISTUTILS_SRC_TEST
The DISTUTILS_SRC_TEST enables support for one of the standard Python test suites. This feature is not supported in distutils-r1 and appropriate python_test sub-phase needs to be written manually. Test suites manual can be consulted for support on running different test suites.

Please remember to add the dependency on the test runner package as well.

Similarly, DISTUTILS_DISABLE_TEST_DEPENDENCY has no counterpart. Its behavior can be reproduced by not adding the fore-mentioned dependency :).