For system and network monitoring, the Nagios tool is a popular choice amongst various free software users. This guide helps you discover Nagios and how you can integrate it with your existing Gentoo infrastructure.
- 1 Introduction
- 2 Setting Up Nagios
- 3 Monitoring nodes
- 4 More Resources
- 5 External resources
Single system users usually don't need a tool to help them identify the state of their system. However, when you have a couple of systems to administer, you will require an overview of your systems' health: do the partitions still have sufficient free space, is your CPU not overloaded, how many people are logged on, are your systems up to date with the latest security fixes, etc.
System monitoring tools, such as the Nagios software we discuss here, offer an easy way of dealing with the majority of metrics you want to know about your system. In larger environments, often called "enterprise environments", the tools aggregate the metrics of the various systems onto a single location, allowing for centralized monitoring management.
The Nagios software is a popular software tool for host, service and network monitoring for Unix (although it can also capture metrics from the Microsoft Windows operating system family). It supports:
- obtaining metrics for local system resources, such as diskspace, CPU usage, memory consumption, ...
- discovering service availability (such as SSH, SMTP and other protocols),
- assuming network outages (when a group of systems that are known to be available on a network are all unreachable),
Basically, the Nagios software consists of a core tool (which manages the metrics), a web server module (which manages displaying the metrics) and a set of plugins (which obtain and send the metrics to the core tool).
About this Document
The primary purpose of this document is to introduce you, Gentoo users, to the Nagios software and how you can integrate it within your Gentoo environment. The guide is not meant to describe Nagios in great detail - I leave this up to the documentation editors of Nagios itself.
Setting Up Nagios
Before you start installing Nagios, draw out and decide which system will become your master Nagios system (i.e. where the Nagios software is fully installed upon and where all metrics are stored) and what kind of metrics you want to obtain. You will not install Nagios on every system you want to monitor, but rather install Nagios on the master system and the TODO on the systems you want to receive metrics from.
Install the Nagios software on your central server:
Follow the instructions the ebuild displays at the end of the installation (i.e. adding
nagios to your active runlevel, configuring web server read access and more).
Really. Read it.
Install plugins on the server as well:
Restricting Access to the Nagios Web Interface
The Nagios web interface allows for executing commands on the various systems monitored by the Nagios plugins. For this purpose (and also because the metrics can have sensitive information) it is best to restrict access to the interface.
For this purpose, we introduce two access restrictions: one on IP level (from what systems can a user connect to the interface) and a basic authentication one (using the username / password scheme).
First, edit /etc/apache2/modules/99_nagios3.conf and edit the
allow from definitions:
Next, create an Apache authorization table where you define the users who have access to the interface as well as their authorizations. The authentication definition file is called .htaccess and contains where the authentication information itself is stored.
Place this file inside the /usr/share/nagios/htdocs and /usr/lib/nagios/cgi-bin directories.
Create the /etc/nagios/auth.users file with the necessary user credentials. By default, the Gentoo nagios ebuild defines a single user called
nagiosadmin . Let's create that user first:
Once Nagios and its dependencies are installed, fire up Apache and Nagios:
Next, fire up your browser and connect to http://localhost/nagios . Log on as the
nagiosadmin user and navigate to the Host Detail page. You should be able to see the monitoring states for the local system.
There are various methods available to monitor nodes.
- Collect data using check_nrpe plugin where NRPE daemon is available on ssh hosts
- Use a password-less SSH connection to execute the command remotely
- Poll hosts with SNMP
- Receive SNMP traps send by nodes and create alerts from it.
Monitoring via NRPE and SNMP are 2 different approaches, a significant difference between NRPE and SNMP monitoring is:
- configuration of NRPE is more node focused, where you actually have privileged access to a monitored node, and define there which checks are interesting for your monitoring system, and allow them.
- configuration of SNMP is more server focused, f.e. where there is no possibility to configure each node, you get only the SNMP community/SNMP user to a certain device, and configure checks on the server side.
Using NRPE method
With NRPE, each remote host runs a daemon (the NRPE deamon) which allows the main Nagios system to query for certain metrics. One can run the NRPE daemon by itself or use an inetd program. I'll leave the inetd method as a nice exercise to the reader and give an example for running NRPE by itself.
First install the NRPE plugin:
Next, edit /etc/nagios/nrpe.cfg to allow your main Nagios system to access the NRPE daemon and customize the installation to your liking. Another important change to the nrpe.cfg file is the list of commands that NRPE supports. For instance, to use
nrpe version 2.12 with Nagios 3, you'll need to change the paths from /usr/nagios/libexec to /usr/lib/nagios/plugins . Finally, launch the NRPE daemon:
Finally, we need to configure the main Nagios system to connect to this particular NRPE instance and request the necessary metrics. To introduce you to Nagios' object syntax, our next section will cover this a bit more throroughly.
Configuring a Remote Host
First, edit /etc/nagios/nagios.cfg and place a
cfg_dir directive. This will tell Nagios to read in all object configuration files in the said directory - in our example, the directory will contain the definitions for remote systems.
Create the directory and start with the first file, nrpe-command.cfg . In this file, we configure a Nagios command called
check_nrpe which will be used to trigger a plugin (identified by the placeholder
$ARG1$ ) on the remote system (identified by the placeholder
$HOSTADDRESS$ ). The
$USER1$ variable is a default pointer to the Nagios installation directory (for instance, /usr/nagios/libexec ).
Next, create a file nrpe-hosts.cfg where we define the remote host(s) to monitor. In this example, we define two remote systems:
Finally, define the service(s) you want to check on these hosts. As a prime example, we run the system load test and disk usage plugins:
That's it. If you now check the service details on the Nagions monitoring site you'll see that the remote hosts are connected and are transmitting their monitoring metrics to the Nagios server.
Using Passwordless SSH Connection
Just as we did by creating the
check_nrpe command, we can create a command that executes a command remotely through a passwordless SSH connection. We leave this up as an interesting exercise to the reader.
A few pointers and tips:
- Make sure the passwordless SSH connection is set up for a dedicated user (definitely not root) - most checks you want to execute do not need root privileges anyway
- Creating a passwordless SSH key can be accomplished with
ssh-keygen, you install a key on the destination system by adding the public key to the .ssh/authorized_keys file
Using SNMP method
With SNMP each node runs a SNMP daemon, not every node will offer SSH and NRPE monitoring, as a example network infrastructure often offers only to monitor via SNMP.
Configure SNMP access on remote nodes as explained in SNMP wiki article. Start SNMP daemon on remote node
Configuring polling server
On server configure /etc/nagios/resources.cfg, add the SNMP community used on remote nodes:
Define a SNMP check for remote hosts, getting the usage of remote hard disk. notice the usage of $USER161$ in that check:
Test the previously defined check, as nagios user
These commands are runs as the nagios system user:
Possible output from example host gentoo, showing a WARNING state, the target rootfilesystem / exceeds the defined warning level -w 80%,
-s shows staticstical or sometimes called performance data which can be used to create MRTG graphs with net-analyzer/pnp4nagios visualizing usage of service parameters over a period of time.
Adding Gentoo Checks
It is quite easy to extend Nagios to include Gentoo-specific checks, such as security checks (GLSAs). Gentoo developer Wolfram Schlich has a
check_glsa.sh script available amongst others.