Troubleshooting

From Gentoo Wiki
Jump to: navigation, search
Other languages:
English

The purpose of this page is to provide users, in particular new ones, with a set of techniques and tools to allow them to troubleshoot and fix problems with their Gentoo setups on their own. In addition, for those problems which are more complicated, this article seeks to provide users with the skills to collect information that will aid them and their supporters in resolving their issues with a greater degree of expediency.

This article assumes that the Gentoo Handbook has been read and that there is a basic understanding of using Gentoo.

Where to get help

Gentoo has quite a few wikis all of which have a wealth of information. Be wary that some articles may contain old or outdated information, but hands down, reading wiki articles is always a critical step when using Gentoo.

On IRC, #gentoo is the recommended channel for general issues. This is probably the fastest way to get an answer.

Since IRC channels require active presence, some users might prefer leaving a message at a forum or [mailing list which continues to exist after closing the browser. When an answer pops up, a notification (usually through an e-mail) is sent back.

When all else fails, and it appears something is broken with Gentoo, Gentoo's bug tracker is a great resource to see if something has already been reported, or to create new bugs if need be. This is a very effective way to reach the Gentoo Developers, which can then use the given feedback to improve the problematic situation and solve the problem.

Programs

This is a collection of tools that are highly recommended:

Package Management

Package Description
app-portage/gentoolkit provides a collection of tools for interacting with Gentoo; it includes the valuable `equery`, `eclean`, `euse` and `eshowkw`, see their man pages for more information.
app-portage/eix provides eix, a tool for querying portage for packages
app-portage/pfl provides e-file, a tool for querying what package contains a given file; it also works for packages that aren't installed since it uses an online lookup.
app-portage/genlop provides genlop, a tool for parsing emerge logs; handy to figure out when certain packages were installed, updated and to see how long they took to compile.
app-portage/elogv provides elogv, a ncurses based interface for browsing emerge logs.

Hardware

Package Description
sys-apps/pciutils provides lspci, a tool for gathering information about PCI devices.
sys-apps/usbutils provides lsusb, a tool for gathering information about USB devices.

Monitoring

Package Description
sys-process/htop provides htop, a tool for monitoring processes. Like top, but more advanced.
sys-process/iotop provides iotop, a top-like tool for monitoring IO activity by process.
net-analyzer/net-top provides net-top, a top-like tool for examining network traffic by protocol, port, and process.

Miscellaneous

Package Description
app-text/wgetpaste provides wgetpaste, a tool for uploading text directly to a pastebin.
net-irc/weechat provides weechat, a simple text based IRC client.

First steps

In order to shape a problem into a solution; first understand the problem well enough. If the problem is not clear then it will be very hard, if not impossible to come up with a solution. An accurate, detailed description of the problem is more likely to yield a solution that best fits the problem.

Identifying the problem

This part may seem like it's a no-brainer, but without it, any troubleshooting will be next to impossible.

Take note of a few things:

  1. Is this problem related to hardware or software?
  2. What was done or has been changed recently that may have been the cause of this issue?
  3. Can additional information about the problem be collected?

Storing any form of detail, logs, other people experiencing it and more can be valuable and gives a good overview.

Hardware issues

Drivers

Driver issues for hardware is one of the more common issues that are reported on IRC.

First and foremost, identify what the hardware is that is causing problems. lspci and lsusb are great tools for this purpose. For demonstration purposes, we'll be looking at an Ethernet card as the troublesome device.

Determining the proper driver

First look with lspci and find the device information:

root #lspci
 03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8071 PCI-E Gigabit Ethernet Controller (rev 16)

It is also possible to use the -n option to give a short-hand notation (which might be easier to search for on the Internet):

root #lspci -n
 03:00.0 0200: 11ab:436b (rev 16)

The model name (0200) and vendor/model (11ab:436b) can be used to look up the device on a place like WikiDevi or Debian HCL to find the right driver to use in Linux.

Verifying the driver is loaded

Getting the driver is less than half of the battle. Most of the trouble with drivers is making sure that they are loaded and operating correctly.

Relying on our friend lspci again, run:

root #lspci -k
 03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8071 PCI-E Gigabit Ethernet Controller (rev 16)
 	Subsystem: Acer Incorporated [ALI] Device 014e
 	Kernel driver in use: sky2

Note that here, an indicator is given that a driver is in use. If not, then no driver has claimed responsibility over this hardware. Assuming the driver was built as a module rather than built-in, try modprobe <modulename> to attempt loading the right module. If the driver was built-in, consider making it a module, because modules can be loaded with modprobe and unloaded with modprobe -r which saves on number reboots while debugging the issue.

Troubleshooting a driver

Compile the driver as a module, then capture the output it generates when the module is loaded. Make sure to first remove all modules that depend on the driver (see lsmod).

The following command will compare the dmesg output after removing the module (in this example, r8169) with the output after adding the module again, effectively showing the messages that were added (lines starting with +).

root #diff -u <(modprobe -r r8169; dmesg) <(sleep 1; modprobe r8169; dmesg) | grep ^+

The most common issue is missing firmware. If a message shows up that informs the administrator about missing firmware, grab either sys-kernel/linux-firmware or a specific firmware package from Portage and install it. After the firmware installation, unload and load the kernel module again to see if this fixes the situation.

Software issues

Portage issues

Dependency graph slot conflicts

Occasionally, messages like this occur when trying to emerge a package:

CODE Dependency graph error
!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

Generally, this means that the packages listed have multiple requested versions in the dependency graph. And fortunately most, if not all, cases allow to resolve it by manually upgrading to the highest requested version. So assuming that the following message was received:

CODE Example of dependency graph error
app-emulation/emul-linux-x86-xlibs:0 
(app-emulation/emul-linux-x86-xlibs-20120127::gentoo, installed) pulled in by 
~app-emulation/emul-linux-x86-xlibs-20120127 required by (app-emulation/emul-linux-x86-medialibs-20120127::gentoo, installed) 
(and 2 more with the same problem) 
 
(app-emulation/emul-linux-x86-xlibs-20120520::gentoo, ebuild scheduled for merge) pulled in by 
>=app-emulation/emul-linux-x86-xlibs-20120520 required by (net-im/skype-2.2.0.35-r99::gentoo, ebuild scheduled for merge) 
~app-emulation/emul-linux-x86-xlibs-20120520 required by (app-emulation/emul-linux-x86-medialibs-20120520::gentoo, ebuild scheduled for merge)

The most recent version desired is 20120520, so:

root #emerge --ask --oneshot =app-emulation/emul-linux-x86-xlibs-20120520

Often, it's advantageous to one-shot (--oneshot or -1) all of the conflicts at once because they sometimes depend on one another.

Here is another example of a block:

CODE Example of a block conflict
app-text/poppler:0
 
 (app-text/poppler-0.24.5::gentoo, installed) pulled in by
   app-text/poppler:0/44=[xpdf-headers(+)] required by (dev-tex/luatex-0.76.0::gentoo, installed)
   app-text/poppler:0/44=[cxx,jpeg,lcms,tiff,xpdf-headers(+)] required by (net-print/cups-filters-1.0.43::gentoo, installed)
 
 (app-text/poppler-0.24.3::gentoo, ebuild scheduled for merge) pulled in by
   >=app-text/poppler-0.12.3-r3:0/43= required by (app-text/texlive-core-2013-r1::gentoo, installed)
   poppler:0/43

There should be no problem changing poppler versions. Note that luatex and cups-filters don't require a specific version of poppler, but they need to be rebuild with whatever version is used:

root #emerge --ask --oneshot =app-text/poppler-0.24.3 dev-tex/luatex net-print/cups-filters

Collecting additional information

When seeking help, more times than never additional information will be asked, such as the output of a command. Some of these produce dozens or hundreds of lines of text - too much to be suitable for pasting into IRC chat rooms, such as the Gentoo support channel on Freenode. That's where wgetpaste comes in handy - it allows users to post short links that contain long output.

Make sure that app-text/wgetpaste is installed:

root #emerge --ask wgetpaste

For instance, to show the contents of /etc/conf.d/net, using wgetpaste is as simple as:

user $wgetpaste /etc/conf.d/net

Or let's say that someone needs the detailed output of an lspci command is:

root #lspci -nnk | wgetpaste

Sometimes it is necessary to redirect stderr to stdout so that error messages can be pasted as well. That can be done like this:

root #emerge -pv 2>&1 | wgetpaste

These return a link to a pastebin site page that can be easily shared with someone to get help.

Things to consider pasting:

  • In case of hardware problems or kernel issues:
user $wgetpaste /usr/src/linux/.config
  • In case of portage issues:
root #emerge --verbose --info | wgetpaste
  • In case of package installation failures:
root #wgetpaste /var/tmp/portage/<category>/<package>-<version>/temp/build.log
  • In case of Xorg issues:
user $wgetpaste /var/log/Xorg.0.log
  • In case all installed packages on the system needs to be provided:
root #eix-update && eix-installed all

Wrapping things together

Below is a shell function that can be used to have several commands executed and their output processed by a single wgetpaste command so that only one URL needs to be provided to the user or developer that wants to help out:

root #
execv(){ for cmd in "$@"; do echo -e "\n=== $cmd ===\n"; $cmd; done; }
(execv "cat /etc/*-release" "uname -a" "lspci -nnk" "lsusb" "eselect profile list"\
 "emerge --info" "cat /usr/src/*$(uname -r)/.config" "cat /var/log/Xorg.0.log" "dmesg"\
 "eix-update" "eix-installed all")\
 | wgetpaste

Please do not use pastebin services on the gentoo-user mailing list

Most pastebin services expire old pastes periodically (not to mention that pastebin services are hardly profitable, and close down all the time) whereas mailing list messages are archived in multiple places and are referenced for years. If the relevant output is short, or can be trimmed, then copy and paste it into the body of the email - longer output can be directed into a text file and added as a plain attachment.

Forming a solution

With a good description of the problem, a complete history of actions and all sorts of debugging data (problem details, hardware / software information, logs, backtraces and more) a good set of useful data is available to look for a solution.

In general, repeat the following steps to come to a solution:

  • Get an idea where the problem might be, think about possible causes.
  • If there are no obvious areas to look into, become more acquainted with the related areas or ask an expert more about it.
  • Sometimes it is necessary to make assumptions to proceed. When assumptions are made, put it to the test: false assumptions should not make users and developers blind of other causes.
  • Obtain additional debugging information in the area of a problem, or perform tests.

Often this will lead to multiple possible causes. It is important to test them (to verify it is the actual cause or not) and therefore test them individually; this is also known as "divide and conquer".

Let's say a problem like "my browser sometimes displays white pages when I boot" comes up and a lot of details are already known about it, then there may be number of possible causes:

  • Is this because the browser was improperly closed when the system rebooted?
Verify this by killing the process and pulling the plug several times, each time checking if the pages are white after booting again. If this is a cause, is it the actual cause or are there other causes?
  • Is this the result of a specific browser version being broken?
Try older or newer versions for a few days or reboots and see if they are broken as well.

And so on...

With the additional information, dare to think about more specific causes:

  • Does a certain pinned tab X that loads plugin Y break the system?
Unpin the tab so it doesn't open on launch anymore or disable the plugin, see if this keeps the problem away.
  • Graphical issues were reported as well, might this maybe mean this is caused by the video drivers?
Try a different version for those drivers, or alternative drivers if available.

As each possible cause is investigated, the actual cause of the problem might come closer and closer.

And in the end, if the cause is still not found, the Where to get help resources at the top of this article yield a multitude of places with people that are looking forward to help; make them happy with a careful problem description and mentioning what has already been tried so far, it'll help them help out.