Project:Python/Tests

This page lists tips and good practices for handling test suites in Python packages.

Determine how to run tests
Every Python package should enable tests if they are provided upstream. In order to enable tests, you need to determine the correct dependencies and test runner. In order to determine those, bear in mind the following advices:


 * 1) Consult the associated documentation. Upstreams sometimes mention how to run tests. However, note that the documentation can be sometimes outdated if upstream changed the test system.
 * 2) Look at common test program/CI configuration files —, . Note that they usually include a lot more than strictly necessary for Gentoo packaging needs; however, they can be useful in figuring out what is actually being done upstream.
 * 3) Look at . Setuptools packages often define test dependencies there (e.g. via tests_require). Some packages may also support running tests via the test command which is usually preferable over calling other test runners directly.
 * 4) Look at imports in test files. The table in test suites section lists common packages providing test suites, and appropriate runners for them.

Do not forget to add appropriate test dependencies. If tests require additional packages, add them.

Skip coverage testing and PEP-8 testing
Most of the tests provided by the packages should be run. However, you can safely omit coverage reports, PEP-8 testing and similar activities that are mostly targeted at the package maintainer and do not serve finding bugs in the package.

Test with network-sandbox
Use to ensure that tests (and the setup script) work correctly without Internet connection. Note that some packages will attempt to fetch missing dependencies over the network — you either need to add those dependencies to ensure that they are installed, or otherwise remove them from.

If some tests require Internet connection, those tests should be disabled to avoid possible data transfer fees on our users. However, please mark this with an explicit comment so that developers willing to run all the tests can re-enable them easily.

Deal with failing tests
If some tests fail, always investigate the issue. You should work with upstream on ensuring that all tests pass correctly. You may disable tests that are clearly broken but ensure that they are re-enabled once the issue is solved.

setuptools (and distutils) test commands
The setuptools build system provides some support for running different test systems via the test command. Furthermore, some packages define their own variants of the test command.

Note that many packages do not use those test commands — and in case of setuptools the test command does not fail and is a no-op. Make sure that any tests are actually run.

Test suites
The following table lists the packages installing common test suites, their module names and packages installing them.

unittest and unittest2
unittest is Python's built-in test suite module. This module is constantly developed and new features are added in every version of Python. Especially, Python 2.7/3.2 have seen many improvements that a number of packages rely on.

unittest2 aims to be a backport of those new features to the older Python versions. In this common case, the affected packages use either unittest2 or unittest conditionally, similarly to the following snippet:

In this case, the developer is responsible for assuring which Python versions require unittest2, and which work fine with plain unittest. Often, requiring as an unconditional dependency is unnecessary.

In Gentoo, we consider two cases:


 * 1) If the tests work fine with plain unittest in all supported implementations, no dependency is used,
 * 2) If the tests require unittest2, dependency on  should be added. It can be specifically versioned and made conditional to some of the Python implementations if only some of the new features are needed.