Munin

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems.

Munin implements a master and slave node structure; one master node with many slave nodes. The master node gathers data from the slave nodes, and produces HTML output. Generally these will be placed in the directory accessible via a web server, and can be viewed with a web browser.

USE flags
The USE flags for Munin are handled in a peculiar way: most of the flags are related to the node, rather than the master (with a few, more or less obvious, exceptions), and are designed to bring in the required dependencies. When the flags are disabled, the plugins are not removed from the package, which means that it's sufficient to just merge the dependencies to get them working, instead of rebuilding Munin. This has been decided because the plugins are generally static, for what concern dependencies.

A notable exception is the java USE flag that actually does remove some plugins. The reason is simply that for the plugins to work, you need a JAR library that is part of Munin, and is thus only built with the flag enabled, so you'd still be rebuilding the package to enable or disable it.

Installation and setup
The installation of Munin is easy enough, as it only requires to install the one package, and contains no extra plugins or anything, by default.

But since Munin is fundamentally made of two components – the node and the master – you should first decide which of the two configurations you want to set up for any given box.

Note: some of the configuration coming up is going to use the 'sudo command. This is required because by default the plugins and the master scripts are executed as the munin user. If you prefer you can easily perform the actions using su instead.

Node (client)
Munin nodes have the task of reporting the results of a series of scripts (called plugins), which report data in a structured format so that the master can log the values and generate a report. Usually you set up one node for each box you intend to monitor, including the master.

To set up a simple node, which only has to gather data and does not have to produce the reports, you're recommended to enable the minimal USE flag — by doing so, you're avoiding a number of extra dependencies that only involve the master, namely. Disabling minimal it's also necessary to disable cgi which is a default starting from version 2.0.2.

The nodes shouldn't in general need any particular configuration specific to Gentoo, so you can follow the upstream manual just fine. In general you can use the munin-node-configure command to enable all the plugins that can be used on a system; this is not recommended though, simply because some of the data might not be of interest and would simply add load to the system when executed.

You can use the following command to identify which plugins would be enabled:

Some of the plugins that are installed might not be functional due to missing dependencies; check the USE flags description for further information.

If the selection sounds correct, you can execute the script by using the following command, or simply manually link the plugins you're interested in.

Finally you have to start up the service and set it for automatic start.

Master
The Munin master has the task of fetching the values reported by the nodes and producing a report, composed of HTML pages and PNG graphs. This can be done either within the cron job (so that the pages are generated statically) or via the CGI scripts that come installed by default. In either case you're likely going to need a web server, such as Apache2, Lighttpd, or nginx, to display the reports.

When setting up a Munin master you have to make sure the minimal USE flag is not enabled.

cron configuration
The Munin master needs to executed periodically a script that gathers the data from the configured nodes; this can be set up in cron by simply using the following command:

This will set up a cron job for the munin user to execute every five minutes. It is designed to work with vixie-cron, dcron and fcron (as of version 2.0.3). The user should already be listed in the cron group required to use the service.

Generating reports
Starting from version 2, it's possible to have both the HTML pages and the graph images generated by CGI scripts. These have been enabled by default by upstream up to version 2.0.5, but due to a number of bugs, including security bugs, these have been disabled by default for both Gentoo and Debian. Upstream is considering disabling CGI by default as well.

For this reason, while the cgi USE flag is present, it's recommended to keep it disabled, and proceed with static genereation of reports, which is equivalent to what was supported in the previous 1.4.x series:

The default configuration will put the generated files in which makes it work out of the box with Apache2 and lighttpd as http://localhost/munin/ — but this might not be suitable or desirable for your setup. If you want to change the path where the files are placed, either to change it to a different virtual host or to a different path, you just have to edit :

SSH Transport and Munin Async
Note! This section discuss Munin async as it's implemented from version 2.0.7-r5 onward. Anything before that is completely messed up and is not worth trying.

Starting from version 2, Munin supports a direct implementation of SSH transport, which allows to gather data from remote nodes that only allow SSH access through the firewall. Using this method allows for people to authenticate all requests.

This is implemented by generating SSH keypairs for the munin user on the master, and accepting them on the nodes for a compatible user. Since the munin user should not be allowed to log-in, as this can induce security issues, it's important to choose a different user to connect to, such as scponly. The Munin ebuild is providing an adequate user in form of munin-async which is also usable for the asynchronous node access.

The public keys to use for allowing access to Munin are generated by emerge --config net-analyzer/munin and are stored as and. You're suggested to use the latter if your SSH supports it. To use the munin-async user, you should copy one or both of them in.

After Munin gets access to the box you have to provide a way for it to contact the Munin node; Munin expects the standard input/output streams of the SSH process to connect straight into the node itself, so the upstream-suggested command to provide is /usr/bin/nc localhost 4949. If you have a recent enough OpenSSH, though, there is a simpler way by using the -W option, by leaving the command section empty:

This will automatically forward input and output to the given address pair without having to execute any other command on the server side. It's quite useful if you have one SSH-able gateway and a number of nodes that are available behind it.

Alternatively it's possible to use the Asynchronous Node. This requires enabling the munin-async service on the node itself, so that it fetches and stores locally the plugins' output, and then sends them to the master when it starts up. For a more thorough description of the Asynchronous Node, please see this post by Flameeyes until more proper documentation is available.

To use the asynchronous node, you just need to provide a slightly different command to the SSH protocol for munin:

The --spoolfetch option provides a batch of updates to the master since last time it connected to that particular node, and helps avoiding blank spaces in the graph itself caused by network congestions.

It's also possible to access an Async node running locally without using SSH, this is achieved by using the cmd:// protocol:

Please note that there are three slashes after the protocol definition, because the hostname has to be kept emtpy.

Integration with Nagios or Icinga
While Munin already provides some degree of warning notification when the nodes' data reaches given thresholds, it doesn't even come near the flexibility of dedicated software such as Nagios or Icinga. Luckily, instead of having to set up the tests twice to provide the data to the two of them, it's possible to make the Munin master forward the problem statuses to Nagios or Icinga through the NSCA software available in.

Gentoo Linux exclusives
There are a few things in Munin that are only available with the Gentoo Linux ebuilds, either because they fit better our design or because upstream is not ready (yet) for them.

In particular as of August 2012, the IPMI plugin shipping with Munin 2.0 and later is not the official one shipped by upstream, but a new one, developed by Flameeyes and based off FreeIPMI. The reason of this change is that the previous plugin, using ipmitool, would time out on most HP and SuperMicro servers when fetching sensors' data.

The new plugin is not available in the official Munin package because it requires a fairly recent FreeIPMI version (much newer than the latest available for Debian and Fedora) even to get near the feature list of the original plugin, while it requires an unreleased, or patched, version to surpass the original plugin. In Portage this is mandated by depending on version 1.1.6-r1 or later (which has been patched to support outputting thresholds, otherwise available only on 1.2.x series).

This IPMI plugin is also able to monitor foreign hosts, and is backward compatible with both ipmi_ and ipmi_sensor_. Finally, it does not require the use of gawk but works with any POSIX-compatible awk implementation.

Troubleshooting
The most common problem is graphs are not generated. Check for problems by executing

Common errors include not restarting  after adding plugins.

External resources

 * Missing graphs
 * Flameeyes's Munin-related posts