The GNU debugger (GDB), is used to find runtime errors that normally involve memory corruption.
USE flags for sys-devel/gdb GNU debugger
||Enable Intel Control-flow Enforcement Technology.|
||Add support for the guile Scheme interpreter|
||Support lzma compression in ELF debug info|
||Support all known targets in one gdb binary|
||Add Native Language Support (using gettextGNU locale utilities)|
||Enable support for the new internal scripting language, as well as extended pretty printers|
||Install the "gdbserver" program (useful for embedded/remote targets)|
||Build gdb's simulators for various hardware platforms. See https://sourceware.org/gdb/wiki/Sim.|
||Enable listing highlighting via dev-util/source-highlight|
||Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)|
||Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes drastically|
||Support parsing XML data files needed (at least) for cpu features, memory maps, and syscall tracing|
||Use dev-libs/xxhash to speed up internal hashing.|
||Enable support for ZSTD compression|
emerge --ask sys-devel/gdb
gdb is a complex program with many alternative interfaces, some optimized for different use cases:
- dev-util/cgdb - terminal UI, ncurses, bit older
- dev-util/gef - terminal UI, just run gef instead
- dev-util/pwndbg - terminal UI, just run pwndbg instead
- dev-util/seer - GUI, quite new
gdb can be configured via the following files:
- ~/.config/gdb/gdbearlyinit which is checked before any other configuration file
- ~/.config/gdb/gdbinit which is the primary configuration file
By default, gdb is quite noisy at startup:
GNU gdb (Gentoo 13.0.90_p20230110 vanilla) 188.8.131.5230110-git Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". (gdb)
To change this, enable
set startup-quietly on
Then afterwards, gdb is far quieter:
By default, gdb does not retain command history or scrollback between runs. This can be changed via the various
set history filename ~/.gdb_history set history save on set history size unlimited set history remove-duplicates unlimited
Retain debug symbols
See Installing debugging information for packages.
In order to debug a program, emerge the program either with
FEATURES="nostrip" to prevent the stripping of debug symbols, or with the needed package.env settings.
Once debug symbols are set, it is time to begin debugging the program.
If more assistance is needed, such as being able to see the code within GDB TUI (GDB Text User Interface), install the dev-util/debugedit package and include
Although not really needed for meaningful backtraces, this significantly helps when actively coding.
Enabling core dumps
Core dumps can also be used for debugging. These core files (named for when RAM was called "core" memory) contain the same information that the program would produce when run with gdb. Here, bad_code is the program that will be debugged with gdb. In order to debug with a core file with bad_code, run gdb ./bad_code core where core is the name of the core file.
Sometimes a program crashes without an easy way to run it manually under gdb. A solution to this can be obtaining a coredump.
If the program's output is as follows, there was no coredump created:
Note that it doesn't say "(core dumped)". If coredumps are already enabled, the following output will appear:
Segmentation fault (core dumped)
To enable core dumps, run the following in a shell:
sysctl kernel.yama.ptrace_scope=0 && echo 2 > /proc/sys/fs/suid_dumpable # Allows running as a normal user
ulimit -c unlimited # Allow any size for coredumps (re-run this as the user running the broken program if needed)
Then running the program again, a corefile should be created:
Segmentation fault (core dumped)
gdb foo foo.core
Reading symbols from foo... Core was generated by `./foo'. #0 0x00007f4fc7b37ae0 in gets () from /usr/lib64/libc.so.6 (gdb) bt #0 0x00007f4fc7b37ae0 in gets () from /usr/lib64/libc.so.6 #1 0x000055d73f62e08f in main () at /tmp/foo.c:10 (gdb)
Note that Systemd users can utilize coredumpctl as well, while OpenRC users may wish to try sys-process/minicoredumper.
Running the program with GDB
Consider building the program with additional warnings.
-Wallprints all compiler warnings. See GCC Debugging Options for details.
Suppose there is a program which needs to be debugged called bad_code. Some person claims that the program crashes and provides an example. Go ahead and test it out:
/bad_code `perl -e 'print "A"x100'`
It seems this person was right. Since the program is obviously broken, there is now a bug at hand. Now, it is time to use gdb to help solve this matter. First, run gdb with the
--args option, then give it the full program with arguments:
gdb --args ./bad_code `perl -e 'print "A"x100'`
GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".
A prompt that says "(gdb)" should display and wait for input. First, run the program with run:
Starting program: /home/chris/bad_code Program received signal SIGSEGV, Segmentation fault. 0xb7ec6dc0 in strcpy () from /lib/libc.so.6
The program should start, and show a notification of SIGSEGV, or Segmentation Fault. This is GDB telling us that the program has crashed. It also gives the last run function it could trace when the program crashes. However, this is not too useful, as there could be multiple strcpy's in the program, making it hard for developers to find which one is causing the issue. In order to help them out, a backtrace can be performed.
A backtrace runs backwards through all the functions that occurred upon program execution, to the function at fault. Functions that return (without causing a crash) will not show up on the backtrace. To get a backtrace at the (gdb) prompt type in bt. Something similar to the following message should be returned:
#0 0xb7ec6dc0 in strcpy () from /lib/libc.so.6 #1 0x0804838c in run_it () #2 0x080483ba in main ()
The trace pattern should be clearly visible.
main() is called first, followed by
run_it(), and somewhere in
run_it() lies the
strcpy() at fault. Things such as this help developers narrow down problems. There are a few exceptions to the output. First off is forgetting to enable debug symbols with
FEATURES="nostrip". With debug symbols stripped, the output looks something like this:
#0 0xb7e2cdc0 in strcpy () from /lib/libc.so.6 #1 0x0804838c in ?? () #2 0xbfd19510 in ?? () #3 0x00000000 in ?? () #4 0x00000000 in ?? () #5 0xb7eef148 in libgcc_s_personality () from /lib/libc.so.6 #6 0x080482ed in ?? () #7 0x080495b0 in ?? () #8 0xbfd19528 in ?? () #9 0xb7dd73b8 in __guard_setup () from /lib/libc.so.6 #10 0xb7dd742d in __guard_setup () from /lib/libc.so.6 #11 0x00000006 in ?? () #12 0xbfd19548 in ?? () #13 0x080483ba in ?? () #14 0x00000000 in ?? () #15 0x00000000 in ?? () #16 0xb7deebcc in __new_exitfn () from /lib/libc.so.6 #17 0x00000000 in ?? () #18 0xbfd19560 in ?? () #19 0xb7ef017c in nullserv () from /lib/libc.so.6 #20 0xb7dd6f37 in __libc_start_main () from /lib/libc.so.6 #21 0x00000001 in ?? () #22 0xbfd195d4 in ?? () #23 0xbfd195dc in ?? () #24 0x08048201 in ?? ()
This backtrace contains a large number of
?? (question) marks. This is because without debug symbols, gdb does not know how the program was run. Hence, it is crucial that debug symbols are not stripped.
Here is what the output looks like with
-ggdb3 flag enabled:
#0 0xb7e4bdc0 in strcpy () from /lib/libc.so.6 #1 0x0804838c in run_it (input=0x0) at bad_code.c:7 #2 0x080483ba in main (argc=1, argv=0xbfd3a434) at bad_code.c:12
Here, a lot more information is available for developers. Not only is function information displayed, but even the exact line numbers of the source files. This method is the most preferred, as long as the extra space can be spared.
The backtrace can be saved to a file by copying and pasting from the terminal. Once gdb tasks are over, quit.
The program is running. Exit anyway? (y or n) y
This ends the walk-through of gdb. However, there are other types of errors such as improper file access that can cause a program to fail during runtime. Find those using strace.
gdb disables ASLR by default to make addresses more predictable, but this can affect how reproducible a bug is. It can mean a bug doesn't occur within gdb in some rare circumstances.
If this happens, try set disable-randomization off within gdb before running the program with r.
- Debugging — general advice on how to enable debugging
- Stack smashing debugging guide — a step-by-step guide to debug stack smashing violations