Icinga

= Introduction = Icinga is a fork of Nagios and is an extremely capable monitoring system. Installation of the basic system is pretty easy but there are lots of really useful add ons. This howto provides a copy n pastable install for many of those addons and integration.

Add hosts and service checks into Icinga itself is not covered but at the end of this install you will have pretty much all the components you could need. Fight with the usage rather than the install.

To make the most of this howto, ensure that the "you should be able to ..." steps actually work before progressing to the next bit. This system is huge and it is easy to get lost.

= Prerequisites = This Howto assumes the following are up and running. Here I use a single vhost configuration file which is made up from all the various module definitions plus a few extras. If you prefer you should be able to follow the modular approach which is the proper Gentoo way.
 * Install Apache including a method of authentication, and ideally SSL encryption
 * Disable the default vhosts in /etc/conf.d/apache2
 * Create a separate vhost in /etc/apache2/vhosts.d/ (see below for a sample config file)
 * Install MySQL, with phpmyadmin (always handy) and a root password configured
 * Consider using my package.use (see below) to get workable USE flags

= Icinga =
 * Install net-analyzer/icinga-1.10.2


 * Add some extras:
 * net-analyzer/nagios-check_ipmi_sensor
 * net-analyzer/nagios-check_mysql_health
 * sys-power/apcupsd nls -cgi -gnome -snmp -usb (this failed to compile for me unless I disabled some USE flags)
 * net-analyzer/nagios-plugins-snmp


 * Add the apache user to the icinga group and restart Apache:


 * Amend the CGI configuration as required. Replace all occurrences of "icingaadmin" in the permission settings with a valid user that the web server authenticates or * which means any authenticated user.


 * Start the daemon


 * At this point Icinga should be up and running and the web interface should be usable at https://monitor.example.co.uk/icinga. The localhost should be monitored and probably showing an error for the http service (its switched off - we are only using https here).

= IDOUtils =


 * Setup the IDO related config


 * Create the DB. The root password is echoed to the screen so watch out for shoulder surfers. A log file is spat out at the end.


 * Start daemons (The Icinga service will startup ico2db itself)


 * Check /var/log/icinga.log that IDO is happy
 * Add to default run level

= Icinga-web =
 * net-analyzer/icinga-web-1.9.0 USE="apache2 mysql pnp -postgres"
 * rrdtool may need media-fonts/dejavu installing


 * Create Database


 * Change /etc/apache2/modules.d/00_default_settings.conf and set servertokens to Minimal and restart Apache


 * Here we are using the web server to authenticate, so add this section to auth.xml. I put it after the default section.  It seems to work fine but more research needed to find out exactly how it works.


 * At this point you should be able to point your browser at: https://monitor.example.co.uk/icinga-web, connect and see localhost being monitored. If there is no data in Icinga-Web then ensure that the ido2db service is running

Admin users
As we are using web server auth, then the root user becomes unavailable. Edit the database directly to enable another admin user:
 * DB: icinga_web, Table: nsm_user. Find the user_id corresponding to the user to "upgrade"
 * In the nsm_user_role table, add two additional entries for that user_id with role_id = 2 and 3
 * Log the user out and in again and the admin menu should appear

= PNP4Nagios = The pnp USE flag for icinga-web pulls in PNP4nagios Bulk Mode with npcdmod - this means minimal extra configuration within Icinga. A module is loaded and that grads the process data and passes it to a daemon called npcd. npcd then proceesses that data and puts it into the rrds for the web interface


 * Create rra config, no need to edit this, defaults are fine


 * Edit npcd config


 * Create a directory for npcdmod which runs from within Icinga to dump its data file into, npcd then gets data from there and processes it


 * Site local changes are made in config_local.php, leave all the other config files alone. On upgrades diff against them for any new parameters and add to the local config as needed.  The ebuild correctly sets icinga perfdata but does not set nagios_base correctly


 * Create the PNP module configuration


 * Amend icinga config to process prefdata


 * Setup the npcd service


 * Restart the icinga service to enable the module and start gathering data


 * Wait for a while for the graphs to be generated and start filling with data - this will take at least five minutes. You should see that npcd is mentioned in /var/log/icinga/icinga.log.  Look for "npcdmod: Ready to run to have some fun!"
 * Icinga-web: Expand the heading on a host or service (little blue square icon with an arrow) there should be a PNP section with links to the graphs.
 * Icinga classic does not have built in support for PNP as such but the extra actions can be configured in a template for this which is not covered here.

Debugging PNP
Debugging pnp can be hard. A script is available to help: cd /usr/src wget http://verify.pnp4nagios.org/verify_pnp_config perl verify_pnp_config -c /etc/icinga/icinga.cfg -m npcdmod -p /etc/pnp/ I think that the script requires you to explicitly state the module using the broker_module parameter within the icinga.cfg, so you may have to convert back to that temporarily to run this test.

= mk-livestatus = The Nagvis developers have deprecated using the ndomy driver and only recommend using mk-livestatus


 * Create this module config file


 * Restart Icinga


 * Check /var/log/icinga/icinga.log for this: "livestatus: Finished initialization. Further log messages go to /var/log/icinga/livestatus.log"

= Nagvis = Nagviz in Portage lags upstream quite badly - its the only package used here that does this. So I use a local ebuild. See below for how to create this. If Portage catches up then ignore that step and install directly.


 * Create config from sample


 * Set some permissions


 * Edit config


 * Remove demo data (put them somewhere for reference)


 * Create a simple map - as you add systems to Icinga, this map will grow with it.


 * Set permissions


 * Test it - https://monitor.example.co.uk/nagvis. You should go straight though using web server authentication and become an admin.  To reset the permissions, you can delete /etc/nagvis/auth.db - it will be recreated from scratch for you. Change logonenvcreaterole in nagvis.ini.cfg to change how future users will be created.


 * Add a cronk to display Nagviz in icinga-web


 * Clear the cache to enable the new cronk

= Icinga Mobile =  Work in progress  (Broken - FIXME: https://icinga.org/display/howtos/Setting+up+Icinga+Mobile)
 * Install a JVM first eg dev-java/oracle-jdk-bin
 * Homepage: https://www.icinga.org/about/icinga-mobile/

cd icinga-mobile autoconf

./configure --with-web-user=apache --with-web-group=apache --with-web-apache-path=/etc/apache2/modules.d --prefix=/usr/local/icinga-mobile make make install make install-apache-config /usr/bin/install -c -b -m 664 -o apache -g apache etc/apache/icinga-mobile.conf /etc/apache2/modules.d/icinga-mobile.conf
 * Generate an API key in Icinga web
 * Edit lib/Model/IcingaConfiguration.js and set the apikey
 * Install the web app


 * Point your browser at https://monitor.example.co.uk/m

= NagiosQL =
 * Download and untar source, patch with SP2.
 * Move to /var/www/localhost/htdocs
 * Symlink to ./nagiosql
 * Set date.timezone in php.ini

Administration -> Settings -> Server protocol -> https
 * Create /etc/icinga/nagiosql and chown icinga:icinga and chmod g+s
 * Temporary: chown apache:apache /var/www/localhost/htdocs/nagiosql/config
 * Run the setup wizard: https:// /nagiosql
 * Set NagiosQL path values:
 * Config: /etc/icinga/nagiosql and /etc/icinga
 * Remove the install directory
 * Temp disable the http -> https redirection so that you can login and amend the config, reload apache. You will need to amend the browser URL to set https:
 * Put the redirector back and reload apache
 * NagiosQL needs to write to this directory - this may need to be fixed every time Icinga is upgraded:

Administration -> Config Targets -> localhost
 * Change Icinga over to using the objects defined in NagiosQL in /etc/icinga/icinga.conf. Set this from within the web GUI itself

= Notes = UPDATE `tbl_host` SET  `alias` = REPLACE(  `alias`,  'A string',  'Another string' )
 * Snags with Pango seemingly needing to access /root/? - reboot fixes it! This manifests itself as graph errors when accessing PNP4Ngios pages
 * Handy SQL - use in phpmyadmin:
 * Find/Replace:
 * Embedded Perl (ePN). If you enable this note that Icinga prefers this to disable it (rather than nagios):
 * 1) icinga: -epn

Sample Apache configuration
The following sample Apache vhost configuration includes settings for mod_auth_kerb and SSL. Replace monitor and example.co.uk and make sure the SSLCert lines correspond to your environment. Note use of SetEnvIf to automatically "login" certain systems with a username where Kerberos fails - handy for automated systems to access this web server without having to be Kerberized. Replace the authentication settings in the Location / section for your environment.

Local ebuild for Nagvis
I cobbled this together from the existing ebuild and it works OK for me.


 * Create a local overlay directory structure


 * Name the repo


 * Set layout


 * Ebuild files


 * Digest the ebuild


 * Add the repo to Portage


 * If you use eix then update its database


 * You should now be able to:

USE flags
These are the flags I generally use and will provide a good starting point

= References =
 * Links to docs for Icinga are provided from within the classic web and icinga web interface