Nagios/Guide

For system and network monitoring, the Nagios tool is a popular choice amongst various free software users. This guide helps you discover Nagios and how you can integrate it with your existing Gentoo infrastructure.

System Monitoring
Single system users usually don't need a tool to help them identify the state of their system. However, when you have a couple of systems to administer, you will require an overview of your systems' health: do the partitions still have sufficient free space, is your CPU not overloaded, how many people are logged on, are your systems up to date with the latest security fixes, etc.

System monitoring tools, such as the Nagios software we discuss here, offer an easy way of dealing with the majority of metrics you want to know about your system. In larger environments, often called "enterprise environments", the tools aggregate the metrics of the various systems onto a single location, allowing for centralized monitoring management.

About Nagios
The Nagios software is a popular software tool for host, service and network monitoring for Unix (although it can also capture metrics from the Microsoft Windows operating system family). It supports:

and more.
 * obtaining metrics for local system resources, such as diskspace, CPU usage, memory consumption, ...
 * discovering service availability (such as SSH, SMTP and other protocols),
 * assuming network outages (when a group of systems that are known to be available on a network are all unreachable),

Basically, the Nagios software consists of a core tool (which manages the metrics), a web server module (which manages displaying the metrics) and a set of plugins (which obtain and send the metrics to the core tool).

About this Document
The primary purpose of this document is to introduce you, Gentoo users, to the Nagios software and how you can integrate it within your Gentoo environment. The guide is not meant to describe Nagios in great detail - I leave this up to the documentation editors of Nagios itself.

Installing Nagios
Before you start installing Nagios, draw out and decide which system will become your master Nagios system (i.e. where the Nagios software is fully installed upon and where all metrics are stored) and what kind of metrics you want to obtain. You will not install Nagios on every system you want to monitor, but rather install Nagios on the master system and the TODO on the systems you want to receive metrics from.

Install the Nagios software on your central server:

Follow the instructions the ebuild displays at the end of the installation (i.e. adding  to your active runlevel, configuring web server read access and more). Really. Read it.

Install plugins on the server as well:

Restricting Access to the Nagios Web Interface
The Nagios web interface allows for executing commands on the various systems monitored by the Nagios plugins. For this purpose (and also because the metrics can have sensitive information) it is best to restrict access to the interface.

For this purpose, we introduce two access restrictions: one on IP level (from what systems can a user connect to the interface) and a basic authentication one (using the username / password scheme).

First, edit and edit the   definitions:

Allow from definitions

Next, create an Apache authorization table where you define the users who have access to the interface as well as their authorizations. The authentication definition file is called and contains where the authentication information itself is stored.

Example .htaccess file

Place this file inside the and  directories.

Create the file with the necessary user credentials. By default, the Gentoo nagios ebuild defines a single user called. Let's create that user first:

Accessing Nagios
Once Nagios and its dependencies are installed, fire up Apache and Nagios:

Next, fire up your browser and connect to http://localhost/nagios. Log on as the  user and navigate to the Host Detail page. You should be able to see the monitoring states for the local system.

Monitoring nodes
There are various methods available to monitor nodes.
 * 1) Collect data using  plugin where NRPE daemon is available on ssh hosts
 * 2) Use a password-less SSH connection to execute the command remotely
 * 3) Poll hosts with SNMP
 * 4) Receive SNMP traps send by nodes and create alerts from it.

Monitoring via NRPE and SNMP are 2 different approaches, a significant difference between NRPE and SNMP monitoring is:
 * configuration of NRPE is more node focused, where you actually have privileged access to a monitored node, and define there which checks are interesting for your monitoring system, and allow them.
 * configuration of SNMP is more server focused, f.e. where there is no possibility to configure each node, you get only the SNMP community/SNMP user to a certain device, and configure checks on the server side.

Using NRPE method
With NRPE, each remote host runs a daemon (the NRPE deamon) which allows the main Nagios system to query for certain metrics. One can run the NRPE daemon by itself or use an inetd program. I'll leave the inetd method as a nice exercise to the reader and give an example for running NRPE by itself.

First install the NRPE plugin:

Next, edit to allow your main Nagios system to access the NRPE daemon and customize the installation to your liking. Another important change to the file is the list of commands that NRPE supports. For instance, to use  version 2.12 with Nagios 3, you'll need to change the paths from  to. Finally, launch the NRPE daemon:

Finally, we need to configure the main Nagios system to connect to this particular NRPE instance and request the necessary metrics. To introduce you to Nagios' object syntax, our next section will cover this a bit more throroughly.

Configuring a Remote Host
First, edit and place a   directive. This will tell Nagios to read in all object configuration files in the said directory - in our example, the directory will contain the definitions for remote systems.

Editing /etc/nagios/nagios.cfg

Create the directory and start with the first file,. In this file, we configure a Nagios command called  which will be used to trigger a plugin (identified by the placeholder   ) on the remote system (identified by the placeholder   ). The  variable is a default pointer to the Nagios installation directory (for instance,  ).

Next, create a file where we define the remote host(s) to monitor. In this example, we define two remote systems:

Finally, define the service(s) you want to check on these hosts. As a prime example, we run the system load test and disk usage plugins:

That's it. If you now check the service details on the Nagions monitoring site you'll see that the remote hosts are connected and are transmitting their monitoring metrics to the Nagios server.

Using Passwordless SSH Connection
Just as we did by creating the  command, we can create a command that executes a command remotely through a passwordless SSH connection. We leave this up as an interesting exercise to the reader.

A few pointers and tips:


 * Make sure the passwordless SSH connection is set up for a dedicated user (definitely not root) - most checks you want to execute do not need root privileges anyway
 * Creating a passwordless SSH key can be accomplished with , you install a key on the destination system by adding the public key to the  file

Using SNMP method
With SNMP each node runs a SNMP daemon, not every node will offer SSH and NRPE monitoring, as a example network infrastructure often offers only to monitor via SNMP.

Configuring nodes
Configure SNMP access on remote nodes as explained in SNMP wiki article. Start SNMP daemon on remote node

Configuring polling server
On server configure, add the SNMP community used on remote nodes:

emerge

Define a SNMP check for remote hosts, getting the usage of remote hard disk. notice the usage of $USER161$ in that check:

Test the previously defined check, as nagios user

These commands are runs as the nagios system user:

Possible output from example host gentoo, showing a WARNING state, the target rootfilesystem / exceeds the defined warning level -w 80%,

-s shows staticstical or sometimes called performance data which can be used to create MRTG graphs with visualizing usage of service parameters over a period of time.

Adding Gentoo Checks
It is quite easy to extend Nagios to include Gentoo-specific checks, such as security checks (GLSAs). Gentoo developer Wolfram Schlich has a  script available amongst others.

External resources

 * the official Nagios website
 * Nagios Exchange, where you can find Nagios plugins and addons
 * Nagios Forge, where developers host Nagios addon projects
 * Nagios Wiki, for more information and tutorials on Nagios
 * PNP4Nagios official website, to create graphs with collected Nagios data