Project:Python/Byte compiling

From Gentoo Wiki
Jump to: navigation, search
Warning
This page is a work in progress by MGorny (talk | contribs). Treat its contents with caution.

Levels of optimization

Python generally supports three levels of optimization:

  • non-optimized code (.pyc),
  • code without debug statements (.pyo, .opt-1.pyc),
  • code without debug and docstrings (.pyo, .opt-2.pyc).

The use of optimization on compiled byte-code depends on implementation:

  • all implementations use plain .pyc for non-optimized code,
  • PyPy2.7 does not write optimized code to byte-code files,
  • Python 2.7 uses .pyo for both levels of optimization, i.e. you can choose only one level and it'll stay,
  • Python 3.5+ and PyPy3.5+ use .opt-1.pyc and .opt-2.pyc for the optimized code.

How to byte-compile Python modules?

Generally, byte-compiling is done via importing the module with appropriate Python optimization flags. Python provides a convenience compileall module to do that for all modules in a directory.

The reference code to perform complete byte-compilation is:

CODE Byte-compiling all modules for all optimization levels
# all
"${PYTHON}" -m compileall -q -f -d "${sitedir}" "${D}${sitedir}"
# python2.7+ & pypy3.5+
"${PYTHON}" -OO -m compileall -q -f -d "${sitedir}" "${D}${sitedir}"
# python3.5+ & pypy3.5+
"${PYTHON}" -O -m compileall -q -f -d "${sitedir}" "${D}${sitedir}"

Note that -d should specify the 'live system' path to the compiled directory, while the position argument takes current (i.e. DESTDIR) path to it.

The first (non-optimized) variant applies to all Python implementations. -OO is done on all but PyPy2.7, -O is done on implementations using separate suffixes for optimization levels (Python3.5+ / PyPy3.5+). Generally, it should be fine to call more variants than necessary but it might be a waste of CPU cycles.

Byte-compilation in build systems

Automake

Automake supports byte-compiling Python modules when installed via PYTHON primary, e.g.:

FILE Makefile.amInstalling a Python module via automake
pyexec_PYTHON = example.py

Note that automake does not byte-compile Python modules for Python 3 correctly: bug #38043 upstream. This is fixed in Gentoo since automake-1.16.1-r2. You need to reconfigure upstream modules to fix that:

FILE foo-1.ebuildFixing py-compile
inherit autotools

DEPEND=">=sys-devel/automake-1.16.1-r2"

src_prepare() {
  default
  eautomake
}