Project:Tinderbox

Tinderbox
Description
Project email	tinderbox@gentoo.org
IRC channel	#gentoo-tinderbox (webchat)
Lead(s)	none No lead election date set
Member(s)	Agostino Sarubbo (ago) tinderbox and CI
Subproject(s) (and inherited member(s))	(none)
Parent Project	Gentoo
Project listing

Note
This page is still a work in progress

What is a tinderbox

tinderbox means a machine that compiles packages to find the most common build and qa issues.

Origins

The 'tinderbox' term, was initially coined by flameeyes that run tinderbox for a lot of years and we take this opportunity to thank him for the idea and the work done.

Other tinderbox projects

At the time of writing, we have two other tinderbox projects that should not be confused with this. See:

Tinderbox-cluster run by zorry
tinderbox run by toralf

Why tinderbox born?

While tinderbox in general is a good stuff, there are two particular reason that made this project go alive:

Because, during Arch Testing / Stabilization, there were a lot of bugs/regressions that nobody reported and they always blocked stabilizations so tinderbox was a way to anticipate those bugs.
Surprisingly, there was a mail and a rant over irc from two different people who moved objection about the quality of tests.

Tinderbox's rules

Following there are the rules at least for tinderbox run by ago:

Bug reports come with a meaningful summary. If there isn't, there wasn't a pattern that match a common error and that happens frequently with src_test() failures. Maintainer is always encouraged to set a proper summary.
Common additional logs (like test-suite.log testlog.txt CMakeOutput.log CMakeError.log LastTest.log config.log testsuite.log autoconf.out) are automatically attached. If you need something else please ask for them.
There may be an internal reference between round brackets on the “Discovered on” line. This is to understand where that failure was reproduced.
If you see ‘ci’ as internal reference after you pushed a fix, it is very probably that the bug still exist, or there is another failure in the same ebuild phase.
The build log contains by default:

The commit of the tree at the time the ebuild was merged, for convenience there is a link
emerge --info
list of the emerge history (qlop -mv)
the list of installed packages (qlist –ICvUSS)

In case of open bugs regarding test failures, CI won't run tests because result in a waste of resources.
When there is an open bug about a package, in case the bug is reproduced with a newer version, a comment on the bug is added and the summary is updated with new new version that reproduces the issue.

What else you need to know:

If you can't reproduce the bug on your system, please try in an empty stage3 (or docker for convenience).
If you see a compressed build log, is because the plain text version exceeds the limits on our bugzilla (1MB).
When you close the bug with a resolution different from RESOLVED/FIXED please not be cryptic.
If in the emerge history there are dev-lang/python-exec, sys-devel/gcc-config and sys-devel/binutils-config at the end, this is a test with USE=”-native-symlinks” against the package.
If you are unsure if your commit fixes the reported issue, please always close the bug (as RESOLVED/TEST-REQUEST if you prefer) so the system will open a new bug in case the problem is still reproducible.
Given the amount of handled bugs it's hard answer to all generic questions. QA warnings are produced by portage so if there is anything unclear you may want to ask publicly (on irc?) to reduce the response time.
Overlays are supported.
Bugs can be filed on github trackers in case Overlays have their own tracker there.
There are kinds of tests that may produce an unstable system (let's says LTO as example). For this reason changes are applied per package instead of globally. That means that while comment #0 of the bug warns about LTO, if the failure is related to a DEPEND of a package that is about to be tested, it probably won't have anything to do with LTO

After get a tinderbox bug

When you get a tinderbox bug, you must always click on links provided by tinderbox, because while you may know the content, in the meantime the wiki content may have been updated with new resources that can help you. When you push a fix, in case of dubious fixes, feel free to close the bug as RESOLVED-> TEST-REQUEST. If the bug is still reproducible you will get another bug with an updated build.log. In case of bugs with tests failures, make sure to use `pkgdev commit -c` or close the bug before pushing the fix. Otherwise, as explained above, `CI` won't run tests.

Tinderbox against a set of packages

You want to test something that involve a set of packages, let's say a new version of dev-java/openjdk, and you want to know how packages behave, you have multiple choices:

Provide a list of packages in a syntax like "app-category/package1 app-category/package2 app-category/package3 ...". That means that bugs will be reported on https://bugs.gentoo.org
Create a github repo that we will treat as an overlay. This repo will contain only the packages you want to test. This scenario is intended when you want to have a preview about how much bugs will be created, so tinderbox will file bugs as github issues and will open a 'fake' issue as app-category/package1 has been build successfully to give the opportunity to store successful build logs for further inspections. An example is here: https://github.com/mgorny/python-bump-testing

Statistics

Per year

2020: 5098 bugs
2021: 4818 bugs
2022: 5817 bugs
2023: 3811 bugs
2024: NaN bugs

Per system

Note
historically, bugs were not filed with the idea of have statistics based on targets, so results may be incomplete.

Goal of this page

It's a matter of fact that in the past tinderbox was done without any written rules. This led to constant discussions, so the goal of this page is to make clear rules about:

What to test
How to test
How to report
What is fine to be reported

Errors in the 'QA Notices'

Tinderbox basically copy-paste the 'QA Notice' into a bug report. If you think that tinderbox reported a wrong 'QA Notice', a false positive 'QA Notice' and so on, you should get in contact with the author of the 'QA Notice' unless it is a custom 'QA Notice' called 'Tinderbox QA Notice'.

Useful bugzilla trackers

This is not a copy paste of all bugzilla trackers but just the most common. The newest one goes at the top.

Summary	Number	Note
sys-devel/gcc-14 porting	914580
sys-devel/autoconf-2.72	893434
missing DEPEND	893356

Future ideas

Here are the things we can test in the future. Test them right now is not a good idea because they will generate a big amount of bugs that need an attention we can't give at this moment.

compiler-rt (sys-devel/clang-common[default-compiler-rt] - sys-devel/clang-toolchain-symlinks[gcc-symlinks] - sys-devel/gcc-config[-native-symlinks])
llvm libc++ (sys-devel/clang-common[default-libcxx] - sys-devel/clang-toolchain-symlinks[gcc-symlinks] - sys-devel/gcc-config[-native-symlinks])
more from app-alternatives/ packages
sys-libs/fortify-headers
Packages that overwrite CFLAGS/CXXFLAGS or add uncommon ones
Packages that compile in src_install()
Packages that miss RDEPEND
Build src_test() with Address Sanitizer