Project:Python/Python.eclass conversion

From Gentoo Wiki
Jump to:navigation Jump to:search

This guide provides tips and guidelines for converting python.eclass packages.

Generic tips

  1. Always bump revision when converting. This is a big enough change, and we'd like to make sure that upgrade wipes out all references to the old eclass.
  2. If in doubt, ask on #gentoo-python (webchat). We're there to help.
  3. Please remember to test the packages after conversion. If the test suite fails the same as before, it's ok, but if the conversion results in new tests failing then likely something went wrong.
  4. Compare the installed file list before and after conversion. Make sure nothing got lost :).
  5. Ensure that Python modules are byte-compiled (.pyc, .pyo). Many packages fail to do that currently.
  6. Many of python.eclass ebuilds are over-complex or even broken. Do not attempt to do 1:1 conversion. Instead, try to improve the ebuild.
  7. Not everything needs to be installed for all Python implementations. python-single-r1 is easier to use, so use it when multi-install isn't necessary. Value your time, we can re-add multiple implementation support later if necessary.

Eclass choice

The following table maps old eclasses to the new ones.

Old eclass Additional conditions New eclass
distutils distutils-r1
python SUPPORT_PYTHON_ABIS set python-r1 (*)
otherwise python-single-r1
twisted twisted-r1
(no eclass) direct build-time dependency on dev-lang/python python-any-r1

(*) If support for multiple implementations is complex and not really required by reverse dependencies, python-single-r1 can be used instead.

python.eclass conversion

Ebuild head

All python.eclass ebuilds start with a few variables that are set before the inherit line. Those are:

  1. PYTHON_DEPEND to specify dependency atom on Python interpreter (using a micro-syntax).
  2. PYTHON_USE_WITH, PYTHON_USE_WITH_OR, PYTHON_USE_WITH_OPT to handle requiring USE flags on the Python interpreter.
  3. SUPPORT_PYTHON_ABIS to enable multi-impl support, and RESTRICT_PYTHON_ABIS to specify opt-out list of supported implementations.

All python-r1 suite eclasses replace those five variables with the following two:

  1. PYTHON_COMPAT to list supported Python implementations (opt-in, bash array).
  2. PYTHON_REQ_USE to specify USE dependency on the Python interpreter (plain USE dependency string).

When deciding which Python implementations to put in PYTHON_COMPAT, please prefer testing over trusting the current values. Additionally check the dependencies. If the dependencies can't support a particular Python implementation, your package must not claim to support it either (otherwise the cross-package dependency couldn't be fulfilled).

The value of RESTRICT_PYTHON_ABIS can be used as a tip; however note that opt-out nature of this variable often means that uncommon implementations such as PyPy or Jython were never tested. If it's not set, look at PYTHON_DEPEND. Remember to remove implementations that are not supported anymore (older than 2.7 and 3.3 respectively). For mapping of old eclass implementation names, look at implementations article.

PYTHON_DEPEND may contain a USE flag name as well. The mapping of this sub-syntax is covered in the ebuild metadata section.

The value of PYTHON_USE_WITH can be usually copied directly to PYTHON_REQ_USE. However, if some of the supported implementations do not support (require) the particular flag, (-) and (+) USE defaults should be appended respectively. A cheat sheet on recommended USE defaults is available in the Implementation USE flags article.

Complex values of PYTHON_USE_WITH as well as uses of PYTHON_USE_WITH_OR and PYTHON_USE_WITH_OPT are not supported.

DIFF Example single-impl ebuild head
python.eclass
EAPI="2"
PYTHON_DEPEND='2'
PYTHON_USE_WITH="ssl"

inherit python
python-single-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 ) # pypy if proven working
PYTHON_REQ_USE="ssl(+)"

inherit python-single-r1
DIFF Example multi-impl ebuild head
python.eclass
EAPI="4"
PYTHON_DEPEND='*'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS="2.6 3.2 2.7-pypy-* 2.7-jython-*"

inherit python
python-r1
EAPI="5"
PYTHON_COMPAT=( python{2_7,3_3,3_4} ) # 3.4 needs to be tested



inherit python-r1

Metadata variables

The RDEPEND and DEPEND variables are filled implicitly by the eclass based on the value of PYTHON_DEPEND variable. If the variable specifies version ranges only, the dependency is added as an unconditional dependency. If it is prefixed by a USE flag name and a question mark (e.g. PYTHON_DEPEND="python? 2"), an appropriate USE flag conditional is added.

In python-r1 suite of eclasses, the ebuild developer is responsible for filling in RDEPEND and DEPEND values manually, as appropriate. The eclasses set PYTHON_DEPS helper variable which contains proper dependency atoms. Depending on the structure of Python dependency, the variable can be either referenced unconditionally in RDEPEND and DEPEND, or placed within a USE conditional block.

Please note that in most cases the Python interpreter dependency needs to be specified both in RDEPEND and in DEPEND. The pure runtime dependency occurs only when no Python references is done during build-time, e.g. when installing trivial scripts without byte-compiling or obtaining interpreter information. The pure build-time dependency is to be combined with python-any-r1 eclass.

The python-r1 and python-single-r1 eclasses add two additional requirements:

  1. All dependencies on other Python packages need to carry the value of PYTHON_USEDEP as a USE dependency.
  2. The value of PYTHON_REQUIRED_USE needs to be referenced in REQUIRED_USE, with USE conditionals matching the ones in RDEPEND and DEPEND.
DIFF Example single-impl ebuild metadata
python.eclass
EAPI="5"
PYTHON_DEPEND='2'

inherit python

RDEPEND="dev-python/foo
    dev-libs/libbar[python]"

DEPEND="${RDEPEND}"
python-single-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 )

inherit python-single-r1

RDEPEND="dev-python/foo[${PYTHON_USEDEP}]
    dev-libs/libbar[python,${PYTHON_USEDEP}]
    ${PYTHON_DEPS}"
DEPEND="${RDEPEND}"
REQUIRED_USE="${PYTHON_REQUIRED_USE}"
DIFF Example single-impl ebuild metadata with conditional use of Python
python.eclass
EAPI="5"
PYTHON_DEPEND='python? 3'

inherit python

RDEPEND="
python? (

    dev-python/foo
    dev-libs/libbar[python]
)"
DEPEND=""
python-single-r1
EAPI="5"
PYTHON_COMPAT=( python3_3 python3_4 )

inherit python-single-r1

RDEPEND="
python? (
    ${PYTHON_DEPS}
    dev-python/foo[${PYTHON_USEDEP}]
    dev-libs/libbar[python,${PYTHON_USEDEP}]
)"
DEPEND="python? ( ${PYTHON_DEPS} )"
REQUIRED_USE="${PYTHON_REQUIRED_USE}"
DIFF Example ebuild metadata for build-time use of Python 2
(no eclass)
EAPI="5"




DEPEND="=dev-lang/python-2*"
python-any-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 )

inherit python-any-r1

DEPEND="${PYTHON_DEPS}"
DIFF Example ebuild metadata for build-time use of Python with USE flags
(no eclass)
EAPI="5"





DEPEND="dev-lang/python
    !dev-lang/python[-threads]"
python-any-r1
EAPI="5"
PYTHON_COMPAT=( python{2_7,3_3,3_4} )
PYTHON_REQ_USE='threads(+)'

inherit python-any-r1

DEPEND="${PYTHON_DEPS}"

pkg_setup

The python.eclass packages usually call the following functions during pkg_setup:

  1. python_set_active_version to enforce a specific Python version in global scope (alike PYTHON_DEPEND),
  2. python_pkg_setup to perform implementation correctness checks and initialize start implementation in multi-impl packages.

In python-r1 suite, the correctness checks are done implicitly through dependencies. The implementation choice is performed by python_setup function which uses the value of PYTHON_COMPAT. However, the function rarely needs to be called directly. Depending on the eclass used:

  1. python-single-r1 and python-any-r1 eclasses export pkg_setup which finds and enable correct Python interpreter globally, if source build is performed. If Python is needed during pkg_{pre,post}* functions when a binary package install is performed, python_setup needs to be called directly instead.
  2. python-r1 does not export pkg_setup. Instead, Python is initialized (locally for the called function) when python_foreach_impl is used. If Python is needed outside those calls, python_setup can be called directly to initialize the environment for best implementation selected.
DIFF Example ebuild setup for single-impl ebuild
python.eclass
EAPI="5"
PYTHON_DEPEND="2"

inherit python

pkg_setup() {
    python_set_active_version 2
    python_pkg_setup
}
python-single-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 )

inherit python-single-r1

pkg_setup() {

    python-single-r1_pkg_setup  # (matches the default)
}
DIFF Example ebuild setup for build-time Python use
(no eclass)
EAPI="5"




pkg_setup() {
    export PYTHON=/usr/bin/python2
}
python-any-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 )

inherit python-any-r1

pkg_setup() {
    python-any-r1_pkg_setup  # (matches the default)
}


Python byte-code compilation

The python.eclass enforced specific rules for Python byte-code compilation:

  1. packages shouldn't have compiled Python modules during build, unless needed for some reason (poorly written tests etc.),
  2. packages mustn't have installed compiled Python modules (.pyc, .pyo files) to the install root,
  3. packages should have compiled modules into byte-code in pkg_postinst, and removed stray byte-code in pkg_postrm.

The python-r1 suite requires quite the opposite: packages need to install byte-compiled modules to the install root.

Therefore, when converting ebuilds:

  1. python_mod_optimize and python_mod_cleanup calls have to be removed, often making pkg_postinst and pkg_postrm unnecessary.
  2. If the ebuild is applying patches or other hacks to disable byte-compilation (such as overwriting py-compile script), those have to be removed.
  3. If the package does not byte-compile installed files even after removing all the hacks, python_optimize function can be used to byte-compile modules.

Please make sure that byte-code is compiled and optimized correctly. This specifically means that:

  1. Python 2.* creates .pyc files in the same directory as .py, while Python 3.* uses a __pycache__ subdirectory.
  2. CPython creates both .pyc and .pyo files, PyPy creates only .pyc.

If .pyc files are not placed in a subdirectory with Python 3, py-compile script may need to be updated (from automake distribution). If .pyo files are not created, python_optimize may need to be used.

DIFF Example module byte-complation with autotools
python.eclass
src_prepare() {
    # disable byte-compilation
    > py-compile
}

pkg_postinst() {
    python_mod_optimize foo
}

pkg_postrm() {
    python_mod_cleanup foo
}
python-r1 or python-single-r1
# just works™
DIFF Example module byte-complation with custom build system
python.eclass
src_install() {
    # custom build system that does not byte-compile
    emake install INSTALL_ROOT="${D}"

}

pkg_postinst() {
    python_mod_optimize foo bar baz
}

pkg_postrm() {
    python_mod_cleanup foo bar baz
}
python-single-r1
src_install() {
    # custom build system that does not byte-compile
    emake install INSTALL_ROOT="${D}"
    python_optimize  # does all packages by default
}
DIFF Example module byte-complation for modules installed out of sitedir
python.eclass
src_install() {
    default

}

pkg_postinst() {
    python_mod_optimize /usr/share/${PN}
}

pkg_postrm() {
    python_mod_cleanup /usr/share/${PN}
}
python-single-r1
src_install() {
    default
    python_optimize "${D}"usr/share/${PN}
}


$(PYTHON), ${EPYTHON}

The python.eclass ebuilds often use various features of the all-in-one $(PYTHON) getter-function. Additionally, in some contexts ${EPYTHON} variable is used to reference the current implementation name.

The python-r1 suite provides two well-defined variables instead: PYTHON and EPYTHON, and appropriate contexts to use them.

python.eclass python-r1 Description Example
$(PYTHON) ${EPYTHON} Executable name of the Python interpreter python3.3
${EPYTHON} ${EPYTHON} Python wrapper / python-exec name for the Python interpreter python3.3
$(PYTHON -a) ${PYTHON} Absolute path to the Python interpreter (used e.g. by autoconf) /usr/bin/python3.3
$(PYTHON -f) python_setup, then ${EPYTHON} Best implementation Python interpreter python2.7
$(PYTHON -a -f) python_setup, then ${PYTHON} Best implementation Python interpreter, abs. path /usr/bin/python2.7
DIFF Example use of PYTHON to run a script
python.eclass
src_compile() {
    default
    cd doc || die
    # note: both absolute and relative path is fine
    $(PYTHON) build_docs.py || die
}
python-single-r1
src_compile() {
    default
    cd doc || die
    # note: both absolute and relative path is fine
    "${PYTHON}" build_docs.py || die
}
DIFF Example use of PYTHON to run a configure script
python.eclass
src_configure() {

    econf PYTHON="$(PYTHON -f)"
}
python-r1
src_configure() {
    python_setup  # PYTHON is exported here
    econf
}

python_convert_shebangs

The python_convert_shebangs function provided by python.eclass is used to change shebangs in Python scripts to a requested value. As a replacement, python-r1 suite provides python_fix_shebang function which updates shebangs to match the requested implementation.

It should be noted that the functions differ in usage. python_convert_shebangs takes a Python version followed by file or directory names. python_fix_shebang takes only path list, and uses the Python version currently in use (set by python_setup or python_foreach_impl). This matches one of the goals of python-r1 suite — always matching Python implementations explicitly and discouraging uses of generic python2 and python3 shebangs.

python_fix_shebang is usually used in python-single-r1 and python-any-r1 only. In multi-impl (python-r1) ebuilds, a few alternatives need to be considered:

  1. if a build system installs a Python script with generic shebang, python_replicate_script can be used to add the script for all supported implementations.
  2. If the script is installed manually, python_doscript can be used inside python_foreach_impl loop to install the script (instead of dobin).

In case of scripts that are used only during build-time, shebang substitution is unnecessary if they use #!/usr/bin/env python[23] shebangs. The eclass ensure that proper wrappers are put in PATH for those executables.

DIFF Example update of shebangs in single-impl ebuild
python.eclass
src_prepare() {
    python_convert_shebangs -r 2 .
}
python-single-r1
src_prepare() {
    python_fix_shebang .
}
DIFF Example handling of scripts in multi-impl ebuild
python.eclass
src_install() {
    default
    # sets generic 'python3' shebang on all files in /usr/bin
    python_convert_shebangs -r 3 "${ED}"usr/bin/*
}
python-r1
src_install() {
    default
    # copies all files in /usr/bin for all enabled impls
    python_replicate_script "${ED}"usr/bin/*
}


python_execute_function

The python_execute_function runs the specified function or command with setup for one or more Python implementations. Depending on the exact use, python-r1 provides two replacements:

  1. python_foreach_impl to execute the function for all enabled Python implementations (the normal behavior),
  2. Project:Python/python-r1#python_setup to setup the environment for the 'best' Python implementation, with the function being called (or inlined) afterwards (replacement for python_execute_function -f).

It should be noted that unlike in python.eclass, python_foreach_impl does not terminate the build if function returns non-zero value (|| return 1 affects only python_foreach_impl return status). If necessary, the function needs to call die explicitly.

PYTHON_CFLAGS

python.eclass provides four flag manipulation variables:

  1. PYTHON_CFLAGS,
  2. PYTHON_CPPFLAGS,
  3. PYTHON_CXXFLAGS,
  4. PYTHON_LDFLAGS.

All those variables support adding or removing flags (+ or - operator, respectively) for Python implementations matching a pattern. This feature is not supported by distutils-r1, and any flag manipulation need to be performed manually.

DIFF Example replacement for PYTHON_CFLAGS
distutils.eclass
PYTHON_CFLAGS=( "2.* + -fno-strict-aliasing" )
distutils-r1
python_compile() {
    if ! python_is_python3; then
        local CFLAGS="${CFLAGS} -fno-strict-aliasing"
    fi
}


distutils.eclass conversion

Divergences from python-r1

distutils-r1 is built on top of python-r1, and therefore most of the instructions for python.eclass apply there as well. However, there are a few notable exceptions:

  1. python-r1 or python-single-r1 is used as a backend, depending on whether DISTUTILS_SINGLE_IMPL is unset or set appropriately.
  2. RDEPEND, DEPEND and REQUIRED_USE are filled in automatically, unless DISTUTILS_OPTIONAL is set.

You can use the two fore-mentioned variables to modify the eclass behavior as necessary for the ported package.

DIFF Example ebuild head for distutils-r1 multi-impl ebuild
distutils.eclass
EAPI="5"
PYTHON_DEPEND='3'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS="2.*"

inherit distutils
distutils-r1
EAPI="5"
PYTHON_COMPAT=( python{3_3,3_4} )



inherit distutils-r1
DIFF Example ebuild head for distutils-r1 single-impl ebuild
distutils.eclass
EAPI="5"
PYTHON_DEPEND='2'


inherit distutils
distutils-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 )
DISTUTILS_SINGLE_IMPL=1

inherit distutils-r1
DIFF Example ebuild for ebuild with optional distutils-r1 part
distutils.eclass
EAPI="5"
PYTHON_DEPEND='python? 2'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS="3.*"

inherit distutils

RDEPEND="
    python? (
        dev-python/foo

    )"
DEPEND=${RDEPEND}


src_compile() {
    default
    
    if use python; then
        cd python || die
        distutils_src_compile
    fi
}
distutils-r1
EAPI="5"
PYTHON_COMPAT=( python2_7 pypy )
DISTUTILS_OPTIONAL=1


inherit distutils-r1

RDEPEND="
    python? (
        dev-python/foo[${PYTHON_USEDEP}]
        ${PYTHON_DEPS}
    )"
DEPEND=${RDEPEND}
REQUIRED_USE="python? ( ${PYTHON_REQUIRED_USE} )"

src_compile() {
    default
    
    if use python; then
        cd python || die
        distutils-r1_src_compile
    fi
}

Phase functions

Both distutils.eclass and distutils-r1 export a few phase functions by default. The following table compares exported functions.

Function distutils.eclass distutils-r1 Behavior
src_unpack In EAPI < 2 No equivalent to default src_unpack() + distutils_src_prepare
src_prepare Yes Yes stubs out ez_setup.py & distribute_setup.py, copies sources for multiple impls, in distutils-r1: additionally applies patches from PATCHES
src_configure No Yes no-op by default
src_compile Yes Yes calls setup.py to build package
src_test Yes Yes distutils.eclass: runs test suite depending on DISTUTILS_SRC_TEST, distutils-r1: no-op by default
src_install Yes Yes calls setup.py to install package, installs documentation from DOCS, in distutils-r1 additionally HTML_DOCS and EXAMPLES
pkg_postinst Yes No byte-compiles installed packages
pkg_postrm Yes No removes stray byte-compiled modules

It should be also noted that distutils.eclass provides opaque phase functions while distutils-r1 splits each phase function into two sub-phases. This has two important implications:

  1. Changing per-impl behavior with distutils.eclass required using python_execute_function and sometimes even inlining parts of the original phase function. In distutils-r1 you can override the sub-phase instead, and call the default implementation inside if necessary.
  2. Complete Python environment setup is available inside the sub-phase functions, reducing the number of necessary eclass function calls and variable declarations. In particular, PYTHONPATH is set properly to reference built Python packages.
DIFF Example documentation build with distutils
distutils.eclass
src_compile() {
    distutils_src_compile
    cd doc || die
    "$(PYTHON -f)" generate-docs.py || die
}

src_install() {
    distutils_src_install
    dodoc -r doc/html
}
distutils-r1
python_compile_all() {

    cd doc || die
    "${PYTHON}" generate-docs.py || die
}

python_install_all() {
    distutils-r1_python_install_all
    dodoc -r doc/html
}
DIFF Example test phase run with distutils
distutils.eclass
src_test() {
    testing() {
    PYTHONPATH=build-${PYTHON_ABI}/lib"$(PYTHON)" test/test_foo.py
    }
    python_execute_function testing
}
distutils-r1
python_test() {

    "${PYTHON}" test/test_foo.py || die "Tests fail with ${EPYTHON}"


}

PYTHON_MODNAME

The PYTHON_MODNAME specified list of packages installed by the package, and caused the respective packages to be compiled in pkg_postinst and cleansed in pkg_postrm. Since the python-r1 suite no longer does byte-compilation in pkg_postinst, and distutils byte-compiles installed Python modules, all references to PYTHON_MODNAME can be removed safely.

DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES

The DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES causes distutils.eclass to create a separate copy of package sources for each Python implementation, and spawn setup.py from those separate copies. The distutils-r1 equivalent variable is called DISTUTILS_IN_SOURCE_BUILD.

DISTUTILS_SETUP_FILES

The DISTUTILS_SETUP_FILES provided support for running setup.py files, possibly multiple, in non-standard paths or using non-standard names. In distutils-r1, those corner cases need to be handled manually.

If setup.py is located in directory other than the default value of ${S}, either S needs to be adjusted or the working directory changed before running distutils-r1_src_* (or distutils-r1_python_*). If multiple setup files need to be run, the fore-mentioned phase function may be called multiple times in different locations.

DISTUTILS_GLOBAL_OPTIONS

The DISTUTILS_GLOBAL_OPTIONS provided support for specifying additional parameters passed to setup.py before the commands. The distutils-r1 equivalent is mydistutilsargs, with the additional recommendation to set it in python_configure_all sub-phase.

DISTUTILS_GLOBAL_OPTIONS provided also support for specifying parameters per-implementation. This is not explicitly supported in distutils-r1. Instead, the ebuild needs to set mydistutilsargs locally as appropriate, e.g. within python_compile() and python_install() calls.

DIFF Example replacement for DISTUTILS_GLOBAL_OPTIONS
distutils.eclass
DISTUTILS_GLOBAL_OPTIONS=( --setuptools )
distutils-r1
python_configure_all() {
    mydistutilsargs=( --setuptools )
}

DISTUTILS_SRC_TEST

The DISTUTILS_SRC_TEST enables support for one of the standard Python test suites. This feature is not supported in distutils-r1 and appropriate python_test() sub-phase needs to be written manually. Test suites manual can be consulted for support on running different test suites.

Please remember to add the dependency on the test runner package as well.

Similarly, DISTUTILS_DISABLE_TEST_DEPENDENCY has no counterpart. Its behavior can be reproduced by not adding the fore-mentioned dependency :).

DIFF Example replacement for DISTUTILS_SRC_TEST
distutils.eclass
DISTUTILS_SRC_TEST=nosetests
inherit distutils
distutils-r1

inherit distutils-r1

DEPEND="test? ( dev-python/nose[${PYTHON_USEDEP}] )"

python_test() {
    nosetests || die "Tests fail with ${EPYTHON}"
}
Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Python/Python.eclass_conversion&oldid=785278"