Project:Python/Tests

From Gentoo Wiki
Jump to: navigation, search

This page lists tips and good practices for handling test suites in Python packages.

Adding test support to Python packages

Every Python package should enable tests if they are provided upstream. In order to enable tests, you need to determine the correct dependencies and test runner. In order to determine those, bear in mind the following advices:

  1. Consult the associated documentation. Upstreams sometimes mention how to run tests. However, note that the documentation can be sometimes outdated if upstream changed the test system.
  2. Look at common test program/CI configuration files — tox.ini, .travis.yml. Note that they usually include a lot more than strictly necessary for Gentoo packaging needs; however, they can be useful in figuring out what is actually being done upstream.
  3. Look at setup.py. Setuptools packages often define test dependencies there (e.g. via tests_require). Some packages may also support running tests via the test command which is usually preferable over calling other test runners directly.
  4. Look at imports in test files. The table in test suites section lists common packages providing test suites, and appropriate runners for them.

Do not forget to add appropriate test dependencies. If tests require additional packages, add them.

Most of the tests provided by the packages should be run. However, you can safely omit coverage reports, PEP-8 testing and similar activities that are mostly targeted at the package maintainer and do not serve finding bugs in the package.

Use FEATURES="network-sandbox" to ensure that tests (and the setup script) work correctly without Internet connection. Note that some packages will attempt to fetch missing dependencies over the network — you either need to add those dependencies to ensure that they are installed, or otherwise remove them from setup.py.

If some tests require Internet connection, those tests should be disabled to avoid possible data transfer fees on our users. However, please mark this with an explicit comment so that developers willing to run all the tests can re-enable them easily.

If some tests fail, always investigate the issue. You should work with upstream on ensuring that all tests pass correctly. You may disable tests that are clearly broken but ensure that they are re-enabled once the issue is solved.

setuptools (and distutils) test commands

The setuptools build system (dev-python/setuptools) provides some support for running different test systems via the test command. Furthermore, some packages define their own variants of the test command.

CODE Example code for running setuptools/distutils-defined tests
python_test() {
  esetup.py test
}

Note that many packages do not use those test commands — and in case of setuptools the test command does not fail and is a no-op. Make sure that any tests are actually run.

Test suites

The following table lists the packages installing common test suites, their module names and packages installing them.

Package name Python package Test runner executable Notes
dev-python/logilab-common logilab.common.test pytest -v -
dev-python/nose nose nosetests -v -
dev-python/pytest pytest or py py.test -v -
dev-python/twisted (was: dev-python/twisted-core) twisted.trial trial -
(built-in) unittest ${PYTHON} -m unittest discover -v since Python 2.7/3.2
dev-python/unittest2 unittest2 unit2.py discover -v see below: unittest and unittest2

unittest and unittest2

unittest is Python's built-in test suite module. This module is constantly developed and new features are added in every version of Python. Especially, Python 2.7/3.2 have seen many improvements that a number of packages rely on.

unittest2 aims to be a backport of those new features to the older Python versions. In this common case, the affected packages use either unittest2 or unittest conditionally, similarly to the following snippet:

CODE Common conditional import of unittest2
try:
    import unittest2 as unittest
except ImportError:
    import unittest

In this case, the developer is responsible for assuring which Python versions require unittest2, and which work fine with plain unittest. Often, requiring dev-python/unittest2 as an unconditional dependency is unnecessary.

In Gentoo, we consider two cases:

  1. If the tests work fine with plain unittest in all supported implementations, no dependency is used,
  2. If the tests require unittest2, dependency on dev-python/unittest2 should be added. It can be specifically versioned and made conditional to some of the Python implementations if only some of the new features are needed.