GCC ICE reporting guide

A Article description::guide to debugging GCC Internal Compiler Errors (ICEs).

This page helps with GCC crashing when building a Gentoo package (or any software) with.

To extract useful information from the crash to report a bug requires a few steps. This article lists the steps and provides a few real-world debugging sessions.

Tl;DR

 * 1) Extract self-contained preprocessed file (extracted with  )
 * 2) Verify  still crashes when ran on preprocessed file
 * 3) Expand   to flags not dependent on your machine
 * 4) File a bug on https://bugs.gentoo.org with the following details:
 * 5) Provide a preprocessed file (or a minimized file)
 * 6) Provide   output
 * 7) Provide
 * 8) Provide   expansion
 * 9) [optional] Minimize amount of flags needed to trigger the error
 * 10) [optional] Minimize self-contained example
 * 11) [optional] File a bug on https://gcc.gnu.org/bugzilla

If you don't know how to all of the above, read on to the detailed guide below!

Rebuilding GCC and glibc with debugging symbols (optional)
To get more detailed crash reports it is suggested to rebuild it with debugging symbols and keeping the full sources. As GCC and glibc are tightly coupled, it is also suggested to do the same for the latter hence you will have more readable traces:


 * Create the following files in the directory :


 * The next step is to activate those settings for, create the following file in :


 * Do the same for The case for, however that latter case depends on if you use  or not. So create one the following files according to your current use case:


 * Finally emerge and   again:

Check if an ICE is reproducible
Sometimes GCC crashes due to external reasons not directly related to GCC:


 * Out-of-memory condition (check )
 * Bad RAM modules
 * Unstable overclock of CPU or RAM

Make sure the above is not your case!

Look at with the ICE and try to resume the build to see if the same command causes the same ICE. Usually the command looks something like .

Go to the  (specified in as well) and rerun the build command.

You should see the exact command and the failure.

Our running example will be an ICE on :

{{Cmd|cd '/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5'|LANG{{=}}C make|output= running build running build_ext INFO: Can't locate Tcl/Tk libs and/or headers building 'cmath' extension gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march=prescott -mfpmath=sse -O2 -pipe -fomit-frame-pointer -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c -o build/temp.linux-i686-3.6/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.o In file included from /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c:11:0: /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h: In function 'cmath_acos': /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h:45:1: internal compiler error: Segmentation fault } ^ Please submit a full bug report, with preprocessed source if appropriate. See  for instructions. }}

The build fails every time  is rerun. Yay!

Extract exact command
If the build system already prints exact command, just re-execute it (make sure you are in the correct directory). If the build system does not print the command, you will have to use other methods of getting command like running  or passing verbose flags manually.

In the case of Python, the command is printed as-is:

{{Cmd|LANG{{=}}C gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march{{=}}prescott -mfpmath{{=}}sse -O2 -pipe -fomit-frame-pointer -fwrapv -std{{=}}c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c -o build/temp.linux-i686-3.6/var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.o|output= In file included from /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c:11:0: /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h: In function 'cmath_acos': /var/tmp/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h:45:1: internal compiler error: Segmentation fault } ^ Please submit a full bug report, with preprocessed source if appropriate. See  for instructions. }}

Extract self-contained source (use -save-temps)
Add  to the  command on the previous step to extract a preprocessed sample. will create  (or   for  ).

Obligatory Python example:

The preprocessed file is in  now. You can get more tips at https://gcc.gnu.org/bugs/.

Expand -march=native, exact gcc version and other system-specific options
Some compiler options like  are dependent on the environment.

You need to resolve them into exact options. One way to do it is to query for expansion. Pick your arch instead of  to avoid diff on  value. That will minimize the diff.

Here  is substituted for.

Make sure bug is reproducible after substitution as well!

Report bug on bugs.gentoo.org
File a bug at https://bugs.gentoo.org/ against toolchain@gentoo.org and provide a few details:


 * output
 * Attach  reproducer

[bonus] minimize needed flags to reproduce failure
You can do similar expansion for optimizer options as well:

To expand  into , you can use

Try to find minimal set of flags needed to trigger the crash by removing some expanded options.

[bonus] minimize self-contained source using cvise
Now to the fun part! Let's shrink  down to a manageable size with help of.

{{Cmd|cvise test.sh cmathmodule.i|output=
 * 1) Write a  tester:
 * 2) Validate tester
 * 3) Reduce!
 * 1) Reduce!

running 4 interestingness tests in parallel

< pass_blank :: 0 >
(1.2 %, 733906 bytes) (27.4 %, 539234 bytes)

< pass_clang_binsrch :: replace-function-def-with-decl >
(29.5 %, 523834 bytes) (30.2 %, 518466 bytes) (32.0 %, 505350 bytes) ...         ******** cmathmodule.i ********

typedef struct { double a } b; b c; d, e, f; g { _setjmp; b h; if (e) h.a = copysign(f, d); c = h; } }}

Done!

contains the same shrunk example.

You can also pass the tester command directly to without creating a shell script, e.g.:

[bonus] Check if bug still exists on vanilla gcc master
Building locally could look like:

Look at configure options of Gentoo's to get more options to add to defaults. Be careful: some configure options like can change code generation enough to trigger or hide a bug.

[bonus] Report bug on gcc.gnu.org/bugzilla
File the bug at https://gcc.gnu.org/bugzilla/ with attached minimal reproducer and  detals.

[bonus] Extract gcc backtrace
Example session would be:

[bonus] Fix gcc bug
By now you have a source tree and a command for how to build it!

A few starting points for you:
 * https://gcc.gnu.org/contribute.html for general notes on coding tips and guidelines
 * https://gcc.gnu.org/onlinedocs/

Good luck!

LTO example
LTO bugs are harder to track down because there's an extra step (a few) between the compiler and the final binary.

GCC's wiki and LinkTimeOptimization touches on this.

PR105600 is an example of such a GCC bug.

Summary
The following example is based on a real case. The first part is to narrow things down to have a minimal set of files then in a second movement, investigate why the error condition is triggered.

Extract the exact compiler command
Whenever the compilation fails with an LTO error, the very first step is to look at what the file contains and search inside for the string   (typically, it just precedes the compiler crash dump):

Did you notice the two double ampersands that enclose the full command line? This is our point of interest, copy what is enclosed in-between and keep it somewhere we will use it later in this section. For now, just change the working directory for the failed package's one somewhere under. Here:

From this vantage point of view:

What we have here is two subdirectories:
 * is where the package source files reside (notably)
 * is where the object files resulting from the compilation process are put (plus some other things we ignore for the moment).

So from the above GCC command line perspective:
 * All files relative paths are relative to
 * All  files relative paths are relative to
 * All files relative paths are relative to

If we give a closer look at the above command line, we can decompose it in three consecutive parts:
 * Part A => spans from  to   this is the various options that governs the GCC's behavior
 * Part B => spans from  to   this big list is all of the object files previouly produced by GCC (we are at the final compilation stage... optimize and link with all other needed objects files and librairies)
 * Part C => spans from  to   which is the list of the libraries/shared object files GCC will link against with.

Notice the presence of a relative path in the middle of part C which is. This file is produced by some previous building steps of the package itself, it will be taken care of a bit later.

Minimize responsible objects
The failure is in which by its nature is considering multiple objects. To narrow down the failure, the full command being passed must be considered.

Let's focus on part B (the big list of  files) of the GCC command-line. The goal is to transform all of those relative paths of this list to their absolute path counterparts. Do you remember the directory structure we have talked about in the previous section herebefore? All of those  files are residing under, thus we must prepend their path with that latter one. As  resides itself under, the full path to prepend with is:  .

Now, create a file (here named ) which contains the whole list of those  files (do not reformat it! just copy/paste the whole list as is). The result should be like:

We will transform to not only convert relative paths to absolute paths but also put one item of the list per line. Here is comes at the rescue (don't forget the final slash in the  directive!):

The file should look like this (one file per line):

Now create a copy of the and name it  (the original contents will be needed later) :

The crunchy part is to use (Package ) to reduce the contents of the list stored in the file  to the bare minimum. The only missing piece of that puzzle is to write a small shell-script (here named ) that will be used by and this where the "part A" and the "part B" (respectively the set of options controlling the compiler's behaviour and the set of libraries to link with) we have talked about in the previous section appears in the landscape.

While "part A" can be kept as is (after all we want the very same options set to reproduce the crash), however, "part B" needs to be changed a little bit. So this:

Becomes that:

The three brought-in changes are:
 * Replacing the original  by   to make GCC consider the object list stored in the file  (rather than the original full-fledged list) and  make GCC output the result to.
 * Substituting the relative path of for its absolute one counterpart (this is an archive produced by the package itself, not something external)
 * Appending  at the very end

Putting all pieces together, the test script is this one:

Now unleash the power of : {{RootCmd 00:00:01 INFO === === 00:00:01 INFO running 16 interestingness tests in parallel 00:00:01 INFO INITIAL PASSES 00:00:01 INFO ===< LinesPass::None >=== 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO ===< LinesPass::None >=== 00:00:02 INFO cache hit for /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad 00:00:02 INFO MAIN PASSES 00:00:02 INFO Termination check: size was 166; now 166 00:00:02 INFO CLEANUP PASSES 00:00:02 INFO ===================== done ====================
 * 1=chmod +x check.sh
 * 2=cvise-delta ./check.sh bad
 * output=

< PASS statistics >
pass name                                             time (s) time (%)   worked   failed  total executed LinesPass::None                                           0.34    12.15        0        0               1

Runtime: 3 seconds Reduced test-cases:

--- /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/temp/bad --- /var/tmp/portage/kde-plasma/xdg-desktop-portal-kde-5.27.4.1-r1/work/xdg-desktop-portal-kde-5.27.4.1_build/src/CMakeFiles/xdg-desktop-portal-kde.dir/xdgshortcut.cpp.o }}

Now the file should contain only a small number of problematic object files:

Manually invoke the command as it is in  from the command-line with that reduced subset should trigger a compiler crash :)

The point being: the same failure now occurs with far fewer arguments because reduced the list down to those responsible.

Now we only need to keep aside:
 * The files and
 * the command-line to use

Perfect! We have a lean work basis. Next step: debug the compiler.

Debugging the compiler
Invoke by passing the whole problematic command-line after  :

Set the variable follow-fork-mode to  ( set follow-fork-mode child ).

External resources

 * Reducing an LTO Linux kernel bug with cvise
 * Bug with an example of needing timeouts and such in cvise script