Showing posts with label icinga-web. Show all posts
Showing posts with label icinga-web. Show all posts

Friday, May 4, 2012

Monitoring tool - init script for icinga, ido2db(idoutils), and npcd(pnp4nagios)

As installing finishes icinga, icinga-web, and pnp4nagios, it's necessary to setup init scripts to run and stop daemon. Of course, each of the source files includes ones, but I prefer a typical format based on RPM package to the ones in the source file. So I modified the init scripts based on RPM packages.

I am going to introduce of  each of the init scripts and verification about how they work.
They are open to the public in my github.
  • daemon and init script
Icinga (based on Nagios RPM package) /etc/init.d/icinga
IDOUtils ( based on NDOUtils RPM package) /etc/init.d/ido2mod
PNP4nagios ( based on Nagios RPM Package a little) /etc/init.d/npcd

icinga

  • create init script based on nagios RPM package
    The patch file is stored here.
# yumdownloader --enablerepo=rpmforge icinga
# mkdir work
# cd work
# rpm2cpio ../ nagios-3.2.3-3.el5.rf.x86_64.rpm | cpid -id ./etc/rc.d/init.d/nagios
# cp etc/rc.d/init.d/nagios ./icinga
# cp icinga{,_diff}
...
# diff -c icinga icinga_diff > icinga.patch
# patch -p0 < icinga.patch
# cp icinga /etc/init.d/icinga
  • start daemon
# /etc/init.d/icinga start
Starting icinga:                                           [  OK  ]
  • stop daemon
# /etc/init.d/icinga stop
Stopping icinga:                                           [  OK  ]
  • restart daemon
# /etc/init.d/icinga restart
Stopping icinga:                                           [  OK  ]
Starting icinga:                                           [  OK  ]
  • condrestart daemon
# /etc/init.d/icinga condrestart
Stopping icinga:                                           [  OK  ]
Starting icinga:                                           [  OK  ]
  • reload daemon
# /etc/init.d/icinga reload
icinga (pid  17359) is running...
Reloading icinga:                                          [  OK  ]
  • check if daemon is running
# /etc/init.d/icinga status
icinga (pid  17359) is running...
  • difference between nagios(rpmpackage) and icinga
# diff -u nagios icinga_diff
--- nagios     2012-05-01 23:34:15.000000000 +0900
+++ icinga_diff        2012-05-03 20:52:17.000000000 +0900
@@ -1,36 +1,38 @@
 #!/bin/sh
 # $Id$
-# Nagios      Startup script for the Nagios monitoring daemon
+# Icinga      Startup script for the Nagios monitoring daemon
 #
 # chkconfig:  - 85 15
-# description:        Nagios is a service monitoring system
-# processname: nagios
-# config: /etc/nagios/nagios.cfg
-# pidfile: /var/nagios/nagios.pid
+# description:        Icinga is a service monitoring system
+# processname: icinga
+# config: /usr/local/icinga/etc/icinga.cfg
+# pidfile: /var/run/icinga.pid
 #
 ### BEGIN INIT INFO
-# Provides:           nagios
+# Provides:           icinga
 # Required-Start:     $local_fs $syslog $network
 # Required-Stop:      $local_fs $syslog $network
-# Short-Description:    start and stop Nagios monitoring server
-# Description:                Nagios is is a service monitoring system
+# Short-Description:    start and stop Icinga monitoring server
+# Description:                Icinga is is a service monitoring system
 ### END INIT INFO

 # Source function library.
 . /etc/rc.d/init.d/functions

-prefix="/usr"
-exec_prefix="/usr"
-exec="/usr/bin/nagios"
-prog="nagios"
-config="/etc/nagios/nagios.cfg"
-pidfile="/var/nagios/nagios.pid"
-user="nagios"
+user="icinga"
+prog="icinga"
+prefix="/usr/local/$prog"
+exec_prefix="${prefix}"
+exec="${prefix}/bin/$prog"
+config="${prefix}/etc/$prog.cfg"
+piddir="/var/run"
+lockdir="/var/lock/subsys"
+pidfile="$piddir/$prog.pid"
+lockfile="${lockdir}/$prog"

+[ -d "$piddir" ] || mkdir -p piddir && chown $prog:$prog $piddir
 [ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

-lockfile=/var/lock/subsys/$prog
-
 start() {
     [ -x $exec ] || exit 5
     [ -f $config ] || exit 6
@@ -47,7 +49,7 @@
     killproc -d 10 $exec
     retval=$?
     echo
-    [ $retval -eq 0 ] && rm -f $lockfile
+    [ $retval -eq 0 ] && rm -f $lockfile $pidfile
     return $retval
 }

@@ -60,7 +62,7 @@
 reload() {
     echo -n $"Reloading $prog: "
     killproc $exec -HUP
-    RETVAL=$?
+    retval=$?
     echo
 }

@@ -70,8 +72,8 @@

 check_config() {
         $nice runuser -s /bin/bash - $user -c "$corelimit >/dev/null 2>&1 ; $exec -v $config > /dev/null 2>&1"
-        RETVAL=$?
-        if [ $RETVAL -ne 0 ] ; then
+        retval=$?
+        if [ $retval -ne 0 ] ; then
                 echo -n $"Configuration validation failed"
                 failure
                 echo
  • about the pidfile and the lockfile path
    Icinga.cfg(also nagios.cfg) defines lockfile as pidfile.
    I'm not sure why they're defined as so, but I think they should be separated.
    I defined he path of pidfile and lockfile in the init script and icinga.cfg
# grep '^lock_file'icinga.cfg
lock_file=/var/run/icinga.pid
# egrep '^(pid|lock)' /etc/init.d/icinga 
piddir="/var/run"
lockdir="/var/lock/subsys"
pidfile="$piddir/$prog.pid"
lockfile="${lockdir}/$prog"

ido2db

  • create init script for ndoutils based on ndo2utils RPM packageThe patch file is stored here.
# yumdownloader --enablerepo=rpmforge ndo2utils
# mkdir work
# cd work
# rpm2cpio ../ndoutils-1.4-0.beta7.3.el5.rf.x86_64.rpm | cpio -id ./etc/init.d/ndoutils
# cp etc/init.d/ndoutils ./ido2db
# cp ido2db{,_diff}
# vi ido2db_diff
...
# diff -c ido2db ido2db_diff > ido2db.patch
# patch -p0 < ido2db.patch
# cp ido2db /etc/init.d/ido2db
  • start daemon
# /etc/init.d/ido2db start
Starting ido2db:                                           [  OK  ]
  • stop daemon
# /etc/init.d/ido2db stop
Stopping ido2db:                                           [  OK  ]
  • restart daemon
# /etc/init.d/ido2db restart
Stopping ido2db:                                           [  OK  ]
Starting ido2db:                                           [  OK  ]
  • condrestart daemon
# /etc/init.d/ido2db condrestart
Stopping ido2db:                                           [  OK  ]
Starting ido2db:                                           [  OK  ]
  • difference between ndo2utils(rpmpackage) and ido2db
# diff ndoutils ndoutils_diff

@@ -1,37 +1,42 @@
 #!/bin/sh
-# Startup script for ndo-daemon
+# Startup script for ido2db-daemon
 #
 # chkconfig: 2345 95 05
-# description: Nagios Database Objects daemon
+# description: Icinga Database Objects daemon

 # Source function library.
 . /etc/rc.d/init.d/functions

-
-BINARY=ndo2db-3x
-DAEMON=/usr/sbin/$BINARY
-CONFIG=/etc/nagios/ndo2db.cfg
-
-[ -f $DAEMON ] || exit 0
-
-prog="ndo2db"
+prog=ido2db
+user=icinga
+prefix=/usr/local/icinga
+exec=$prefix/bin/$prog
+config=$prefix/etc/ido2db.cfg
+piddir="/var/run"
+lockdir="/var/lock/subsys"
+pidfile="$piddir/$prog.pid"
+lockfile="${lockdir}/$prog"

 start() {
+    [ -x $exec ] || exit 5
+    [ -f $config ] || exit 6
     echo -n $"Starting $prog: "
-    daemon --user nagios $DAEMON -c $CONFIG
-    RETVAL=$?
+    daemon --user $user $exec -c $config
+    retval=$?
+    [ $retval -eq 0 ] && touch $lockfile
     echo
-    return $RETVAL
+    return $retval
 }

 stop() {
-    if test "x`pidof $BINARY`" != x; then
+    if test "x`pidof $prog`" != x; then
         echo -n $"Stopping $prog: "
-        killproc ndo2db-3x
+        killproc $prog
         echo
     fi
-    RETVAL=$?
-    return $RETVAL
+    retval=$?
+    [ $retval -eq 0 ] && rm -f $lockfile $pidfile
+    return $retval
 }

 case "$1" in
@@ -44,14 +49,14 @@
             ;;

         status)
-            status $BINARY
+            status $prog
             ;;
         restart)
             stop
             start
             ;;
         condrestart)
-            if test "x`pidof $BINARY`" != x; then
+            if test "x`pidof $prog`" != x; then
                 stop
                 start
             fi
@@ -63,5 +68,5 @@

 esac

-exit $RETVAL
+exit $retval
  • about the pidfile and the lockfile path
    Icinga.cfg(also nagios.cfg) defines lockfile as pidfile.
    I'm not sure why they're defined as so, but I think they should be separated.
    I defined he path of pidfile and lockfile in the init script and icinga.cfg
# grep '^lock_file'ido2db.cfg
lock_file=/var/run/ido2db.pid
# egrep '^(pid|lock)' /etc/init.d/icinga 
piddir="/var/run"
lockdir="/var/lock/subsys"
pidfile="$piddir/$prog.pid"
lockfile="${lockdir}/$prog"


npcd

  • create init script for npcd based on nagios RPM packageThe patch file is stored here.
# yumdownloader --enablerepo=rpmforge icinga
# mkdir work
# cd work
# rpm2cpio ../ nagios-3.2.3-3.el5.rf.x86_64.rpm | cpid -id ./etc/rc.d/init.d/nagios
# cp etc/rc.d/init.d/nagios ./npcd
# cp npcd{,_diff}
...
# diff -c npcd npcd_diff > npcd.patch
# patch -p0 < npcd.patch
# cp npcd /etc/init.d/npcd
  • start daemon
# /etc/init.d/npcd start
npcd is stopped
Starting npcd:                                             [  OK  ]
  • stop daemon
# /etc/init.d/npcd stop
npcd (pid  14128) is running...
Stopping npcd:                                             [  OK  ]
  • restart daemon
# /etc/init.d/npcd restart
Starting npcd:                                             [  OK  ]
Starting npcd:                                             [  OK  ]
  • condrestart daemon
# /etc/init.d/npcd condrestart
npcd (pid  14216) is running...
Stopping npcd:                                             [  OK  ]
Starting npcd:                                             [  OK  ]
  • reload daemon
# /etc/init.d/npcd reload
npcd (pid  14233) is running...
Reloading npcd:                                            [  OK  ]
  • check if daemon is running
# /etc/init.d/npcd status
 npcd (pid 14233) is running...
  • difference between nagios(rpmpackage) and npcd
# diff -u npcd npcd_diff
--- npcd       2012-05-04 10:47:11.000000000 +0900
+++ npcd_diff  2012-05-03 22:45:28.000000000 +0900
@@ -1,41 +1,40 @@
 #!/bin/sh
-# $Id$
-# Nagios      Startup script for the Nagios monitoring daemon
-#
-# chkconfig:  - 85 15
-# description:        Nagios is a service monitoring system
-# processname: nagios
-# config: /etc/nagios/nagios.cfg
-# pidfile: /var/nagios/nagios.pid
 #
 ### BEGIN INIT INFO
-# Provides:           nagios
-# Required-Start:     $local_fs $syslog $network
-# Required-Stop:      $local_fs $syslog $network
-# Short-Description:    start and stop Nagios monitoring server
-# Description:                Nagios is is a service monitoring system
+# Short-Description: pnp4nagios NPCD Daemon Version 0.6.16
+# Description: Nagios Performance Data C Daemon
+# chkconfig: 345 99 01
+# processname: npcd
+# config: /usr/local/pnp4nagios/etc/npcd.cfg
+# pidfile: /var/run/npcd.pid
+# Provides:          npcd
+# Required-Start:
+# Required-Stop:
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
 ### END INIT INFO

 # Source function library.
 . /etc/rc.d/init.d/functions

-prefix="/usr"
-exec_prefix="/usr"
-exec="/usr/bin/nagios"
-prog="nagios"
-config="/etc/nagios/nagios.cfg"
-pidfile="/var/nagios/nagios.pid"
-user="nagios"
+user="icinga"
+prog="npcd"
+prefix="/usr/local/pnp4nagios"
+exec_prefix="${prefix}"
+exec="${prefix}/bin/$prog"
+config="${prefix}/etc/$prog.cfg"
+piddir="/var/run"
+lockdir="/var/lock/subsys"
+pidfile="/var/run/$prog.pid"
+lockfile="${lockdir}/$prog"

 [ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

-lockfile=/var/lock/subsys/$prog
-
 start() {
     [ -x $exec ] || exit 5
     [ -f $config ] || exit 6
     echo -n $"Starting $prog: "
-    daemon --user=$user $exec -d $config
+    daemon --user=$user $exec -d -f $config
     retval=$?
     echo
     [ $retval -eq 0 ] && touch $lockfile
@@ -47,7 +46,7 @@
     killproc -d 10 $exec
     retval=$?
     echo
-    [ $retval -eq 0 ] && rm -f $lockfile
+    [ $retval -eq 0 ] && rm -f $lockfile $pidfile
     return $retval
 }

@@ -60,31 +59,14 @@
 reload() {
     echo -n $"Reloading $prog: "
     killproc $exec -HUP
-    RETVAL=$?
+    retval=$?
     echo
 }

-force_reload() {
-    restart
-}
-
-check_config() {
-        $nice runuser -s /bin/bash - $user -c "$corelimit >/dev/null 2>&1 ; $exec -v $config > /dev/null 2>&1"
-        RETVAL=$?
-        if [ $RETVAL -ne 0 ] ; then
-                echo -n $"Configuration validation failed"
-                failure
-                echo
-                exit 1
-
-        fi
-}
-

 case "$1" in
     start)
         status $prog && exit 0
-      check_config
         $1
         ;;
     stop)
@@ -92,33 +74,21 @@
         $1
         ;;
     restart)
-      check_config
         $1
         ;;
     reload)
         status $prog || exit 7
-      check_config
         $1
         ;;
-    force-reload)
-      check_config
-        force_reload
-        ;;
     status)
         status $prog
         ;;
-    condrestart|try-restart)
+    condrestart)
         status $prog|| exit 0
-      check_config
         restart
         ;;
-    configtest)
-        echo -n  $"Checking config for $prog: "
-        check_config && success
-        echo
-      ;;
     *)
-        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
+        echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}"
         exit 2
 esac
 exit $?


I will list the other configurations for icinga, idoutils, and pnp4nagios next time.

Monitoring tool - setup icinga-web with pnp4nagios

As introducing how to install Icinga and Icinga-web before, I am trying to install pnp4nagios and setup icinga-web with pnp4nagios. As Nagios and Icinga themselves have no graphing function, some may use Cacti or Munin for graphing performance data.  There's an addon called PNP4nagios released in Nagios Addon Project, which analyzes performance data provided by nagios-plugins and stores them automatically into RRD-databases.




Then, let's see how to install pnp4nagios and integrate it into Icinga-web. 

install pnp4nagios

  • install rrdtool, perl-rrdtool
# rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
# rpm -ivh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
# sed -i 's|enabled\s\?=\s\?1|enabled = 0|' /etc/yum.repos.d/rpmforge.repo
# yum -y install pango.x86_64 pango-devel.x86_64 glib2.x86_64 glib2-devel.x86_64
# yum -y --enablerepo=rpmforge install rrdtool perl-rrdtool  
  • install pnp4nagios
$ curl http://downloads.sourceforge.net/project/pnp4nagios/PNP-0.6/pnp4nagios-0.6.16.tar.gz | tar zx
$ cd pnp4nagios-0.6.16
./configure  \
--prefix=/usr/local/pnp4nagios-0.6.16  \
--with-nagios-user=icinga \
--with-nagios-group=icinga \
--with-rrdtool \
--with-httpd-conf=/usr/local/httpd/conf/extra \
--with-init-dir=/etc/init.d \
--with-perfdata-logfile=/var/log/icinga
$ make all
# make install install-webconf install-config install-init
# ln -s /usr/local/pnp4nagios-0.6.16 /usr/local/pnp4nagios


Before setting up the configuration, I'll explain the art of collecting data of pnp4nagios. Pnp4nagios has 5 kinds of ways to collect data, synchronous-mode, bulk mode, bulk mode with NPCD, bulk mode with npcdmod, and gearman mode. The detail about each way is shown, here. I adopt bulk mode with npcdmod, but I am also showing how to setup bulk mode with NPCD.

setup bulk mode with NPCD

  • setup configuration files
# cd /usr/local/pnp4nagios/etc/
# mv npcd.cfg-sample npcd.cfg
# mv process_perfdata.cfg-sample process_perfdata.cfg
# mv rra.cfg-sample rra.cfg
  • setup icinga.cfg
# cat >> /usr/local/icinga/etc/icinga.cfg << EOF
## Bulk / NPCD mode
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::\$TIMET\$\tHOSTNAME::\$HOSTNAME\$\tSERVICEDESC::\$SERVICEDESC\$\tSERVICEPERFDATA::\$SERVICEPERFDATA
\$\tSERVICECHECKCOMMAND::\$SERVICECHECKCOMMAND\$\tHOSTSTATE::\$HOSTSTATE\$\tHOSTSTATETYPE::\$HOSTSTATETYPE\$\tSERVICESTATE::\$SERVICESTATE\$\tSERVICESTATETYPE::\$SERVI
CESTATETYPE\$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file

# host performance data starting with Nagios 3.0
host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::\$TIMET\$\tHOSTNAME::\$HOSTNAME\$\tHOSTPERFDATA::\$HOSTPERFDATA\$\tHOSTCHECKCOMMAND::\$HOSTCHECKCOMMAND\$\
tHOSTSTATE::\$HOSTSTATE\$\tHOSTSTATETYPE::\$HOSTSTATETYPE\$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file
EOF
  • setup commands.cfg
# cat >> /usr/local/icinga/etc/objects/commands.cfg << EOF
## Bulk with NPCD mode
define command{
     command_name    process-service-perfdata-file
     command_line    /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.\$TIMET\$
}

define command{
     command_name    process-host-perfdata-file
     command_line    /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.\$TIMET\$
}
EOF
  • start apache, npcd, and icinga daemon
# /etc/init.d/httpd start
# /etc/init.d/npcd start
# /etc/init.d/icinga restart
  • check the configuration settings and performance data
    It passes without critical error.
# wget http://verify.pnp4nagios.org/verify_pnp_config
# perl verify_pnp_config -m bulk+npcd -c /usr/local/icinga/etc/icinga.cfg -p /usr/local/pnp4nagios/etc/
[INFO]  ========== Starting Environment Checks ============
[INFO]  My version is: verify_pnp_config-0.6.17-R.33
[INFO]  Reading /usr/local/icinga/etc/icinga.cfg
[OK  ]  Running product is 'icinga'
[OK  ]  object_cache_file is defined
[OK  ]  object_cache_file=/usr/local/icinga/var/objects.cache
[INFO]  Reading /usr/local/icinga/var/objects.cache
[OK  ]  resource_file is defined
[OK  ]  resource_file=/usr/local/icinga/etc/resource.cfg
[INFO]  Reading /usr/local/icinga/etc/resource.cfg
[INFO]  Reading /usr/local/pnp4nagios/etc//process_perfdata.cfg
[OK  ]  No pnp4nagios_release file found. This might be an older version of PNP4Nagios
[OK  ]  Effective User is 'icinga'
[OK  ]  User icinga exists with ID '1002'
[OK  ]  Effective group is 'icinga'
[OK  ]  Group icinga exists with ID '1002'
[INFO]  ========== Checking Bulk Mode + NPCD Config  ============
[OK  ]  process_performance_data is 1 compared with '/1/'
[OK  ]  service_perfdata_file is defined
[OK  ]  service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata
[OK  ]  service_perfdata_file_template is defined
[OK  ]  service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
[OK  ]  PERFDATA template looks good
[OK  ]  service_perfdata_file_mode is defined
[OK  ]  service_perfdata_file_mode=a
[OK  ]  service_perfdata_file_processing_interval is defined
[OK  ]  service_perfdata_file_processing_interval=15
[OK  ]  service_perfdata_file_processing_command is defined
[OK  ]  service_perfdata_file_processing_command=process-service-perfdata-file
[OK  ]  host_perfdata_file is defined
[OK  ]  host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
[OK  ]  host_perfdata_file_template is defined
[OK  ]  host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
[OK  ]  PERFDATA template looks good
[OK  ]  host_perfdata_file_mode is defined
[OK  ]  host_perfdata_file_mode=a
[OK  ]  host_perfdata_file_processing_interval is defined
[OK  ]  host_perfdata_file_processing_interval=15
[OK  ]  host_perfdata_file_processing_command is defined
[OK  ]  host_perfdata_file_processing_command=process-host-perfdata-file
[INFO]  Icinga config looks good so far
[INFO]  ========== Checking config values ============
[OK  ]  npcd daemon is running
[OK  ]  /usr/local/pnp4nagios/etc/npcd.cfg is used by npcd and readable
[INFO]  Reading /usr/local/pnp4nagios/etc/npcd.cfg
[OK  ]  perfdata_spool_dir is defined
[OK  ]  perfdata_spool_dir=/usr/local/pnp4nagios/var/spool/
[OK  ]  Command process-service-perfdata-file is defined
[OK  ]  '/bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$'
[OK  ]  Command looks good
[OK  ]  Command process-host-perfdata-file is defined
[OK  ]  '/bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$'
[OK  ]  Command looks good
[OK  ]  Script /usr/local/pnp4nagios/libexec/process_perfdata.pl is executable
[INFO]  ========== Starting global checks ============
[OK  ]  status_file is defined
[OK  ]  status_file=/usr/local/icinga/var/status.dat
[INFO]  Reading /usr/local/icinga/var/status.dat
[INFO]  ==== Starting rrdtool checks ====
[OK  ]  RRDTOOL is defined
[OK  ]  RRDTOOL=/usr/bin/rrdtool
[OK  ]  /usr/bin/rrdtool is executable
[OK  ]  RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi@oetiker.ch>
[OK  ]  USE_RRDs is defined
[OK  ]  USE_RRDs=1
[WARN]  Perl RRDs modules are not loadable
[INFO]  ==== Starting directory checks ====
[OK  ]  RRDPATH is defined
[OK  ]  RRDPATH=/usr/local/pnp4nagios/var/perfdata
[OK  ]  Perfdata directory '/usr/local/pnp4nagios/var/perfdata' exists
[WARN]  7 hosts/services are not providing performance data
[WARN]  'process_perf_data 1' is set for 8 hosts/services which are not providing performance data!
[OK  ]  'process_perf_data 1' is set for 34 of your hosts/services
[INFO]  ==== System sizing ====
[OK  ]  33 hosts/service objects defined
[INFO]  ==== Check statistics ====
[WARN]  Warning: 3, Critical: 0
[WARN]  Checks finished...

setup bulk mode with npcdmod

  • setup configuration files
# cd /usr/local/pnp4nagios/etc/
# mv npcd.cfg-sample npcd.cfg
# mv process_perfdata.cfg-sample process_perfdata.cfg
# mv rra.cfg-sample rra.cfg
  • setup icinga.cfg
# cat >> /usr/local/icinga/icinga.cfg << EOF
process_performance_data=1
broker_module=/usr/local/pnp4nagios/lib/npcdmod.o config_file=/usr/local/pnp4nagios/etc/npcd.cfg
EOF
  • start apache, npcd, and icinga
# /etc/init.d/httpd start
# /etc/init.d/npcd start
# /etc/init.d/icinga start
  • icinga.log after enabling the ndomod
# tail -f /var/log/icinga/icinga.log
[1335878074] Caught SIGTERM, shutting down...
[1335878074] Successfully shutdown... (PID=21017)
[1335878074] idomod: Shutdown complete.
[1335878074] Event broker module '/usr/local/icinga/bin/idomod.o' deinitialized successfully.
[1335878074] npcdmod: If you don't like me, I will go out! Bye.
[1335878074] Event broker module '/usr/local/pnp4nagios/lib/npcdmod.o' deinitialized successfully.
[1335878075] Icinga 1.6.1 starting... (PID=21253)
[1335878075] Local time is Tue May 01 22:14:35 JST 2012
[1335878075] LOG VERSION: 2.0
[1335878075] idomod: IDOMOD 1.6.1 (12-02-2011) Copyright(c) 2005-2008 Ethan Galstad, Copyright(c) 2009-2011 Icinga Development Team (https://www.icinga.org)
[1335878075] idomod: Successfully connected to data sink.  0 queued items to flush.
[1335878075] Event broker module '/usr/local/icinga/bin/idomod.o' initialized successfully.
[1335878075] npcdmod: Copyright (c) 2008-2009 Hendrik Baecker (andurin@process-zero.de) - http://www.pnp4nagios.org
[1335878075] npcdmod: /usr/local/pnp4nagios/etc/npcd.cfg initialized
[1335878075] npcdmod: spool_dir = '/usr/local/pnp4nagios/var/spool/'.
[1335878075] npcdmod: perfdata file '/usr/local/pnp4nagios/var/perfdata.dump'.
[1335878075] npcdmod: Ready to run to have some fun!
[1335878075] Event broker module '/usr/local/pnp4nagios/lib/npcdmod.o' initialized successfully.
[1335878075] Finished daemonizing... (New PID=21256)
[1335878076] Event loop started...
  • check the configuration settings and performance data
# wget http://verify.pnp4nagios.org/verify_pnp_config
# perl verify_pnp_config -m npcdmod -c /usr/local/icinga/etc/icinga.cfg -p /usr/local/pnp4nagios/etc/
[INFO]  ========== Starting Environment Checks ============
[INFO]  My version is: verify_pnp_config-0.6.17-R.33
[INFO]  Reading /usr/local/icinga/etc/icinga.cfg
[OK  ]  Running product is 'icinga'
[OK  ]  object_cache_file is defined
[OK  ]  object_cache_file=/usr/local/icinga/var/objects.cache
[INFO]  Reading /usr/local/icinga/var/objects.cache
[OK  ]  resource_file is defined
[OK  ]  resource_file=/usr/local/icinga/etc/resource.cfg
[INFO]  Reading /usr/local/icinga/etc/resource.cfg
[INFO]  Reading /usr/local/pnp4nagios/etc//process_perfdata.cfg
[OK  ]  No pnp4nagios_release file found. This might be an older version of PNP4Nagios
[OK  ]  Effective User is 'icinga'
[OK  ]  User icinga exists with ID '1002'
[OK  ]  Effective group is 'icinga'
[OK  ]  Group icinga exists with ID '1002'
[INFO]  ========== Checking npcdmod Mode Config  ============
[OK  ]  process_performance_data is 1 compared with '/1/'
[OK  ]  event_broker_options is defined
[OK  ]  event_broker_options=-1
[OK  ]  event_broker_option bits 2 and 3 enabled (12)
[OK  ]  broker_module is defined
[OK  ]  broker_module=/usr/local/pnp4nagios/lib/npcdmod.o config_file=/usr/local/pnp4nagios/etc/npcd.cfg
[OK  ]  npcdmod.o config file is /usr/local/pnp4nagios/etc/npcd.cfg
[OK  ]  /usr/local/pnp4nagios/etc/npcd.cfg used by npcdmod.o is readable
[OK  ]  npcd daemon is running
[OK  ]  /usr/local/pnp4nagios/etc/npcd.cfg is used by npcd and readable
[OK  ]  npcd and npcdmod.o are using the same config file (/usr/local/pnp4nagios/etc/npcd.cfg)
[INFO]  Icinga config looks good so far
[INFO]  ========== Checking config values ============
[INFO]  Reading /usr/local/pnp4nagios/etc/npcd.cfg
[OK  ]  Script /usr/local/pnp4nagios/libexec/process_perfdata.pl is executable
[INFO]  ========== Starting global checks ============
[OK  ]  status_file is defined
[OK  ]  status_file=/usr/local/icinga/var/status.dat
[INFO]  Reading /usr/local/icinga/var/status.dat
[INFO]  ==== Starting rrdtool checks ====
[OK  ]  RRDTOOL is defined
[OK  ]  RRDTOOL=/usr/bin/rrdtool
[OK  ]  /usr/bin/rrdtool is executable
[OK  ]  RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi@oetiker.ch>
[OK  ]  USE_RRDs is defined
[OK  ]  USE_RRDs=1
[WARN]  Perl RRDs modules are not loadable
[INFO]  ==== Starting directory checks ====
[OK  ]  RRDPATH is defined
[OK  ]  RRDPATH=/usr/local/pnp4nagios/var/perfdata
[OK  ]  Perfdata directory '/usr/local/pnp4nagios/var/perfdata' exists
[WARN]  7 hosts/services are not providing performance data
[WARN]  'process_perf_data 1' is set for 8 hosts/services which are not providing performance data!
[OK  ]  'process_perf_data 1' is set for 34 of your hosts/services
[INFO]  ==== System sizing ====
[OK  ]  33 hosts/service objects defined
[INFO]  ==== Check statistics ====
[WARN]  Warning: 3, Critical: 0
[WARN]  Checks finished...

Integrate pnp4nagios into icinga-web

  • integrate XML extension
# cp /usr/local/src/icinga-web-1.6.2/contrib/PNP_Integration/templateExtensions/* \
/usr/local/icinga-web/app/modules/Cronks/data/xml/extensions/
  • clear cache
# /usr/local/icinga-web/bin/clearcache.sh
Basedir: /usr/local/icinga-web-1.6.2 Cachedir: /usr/local/icinga-web-1.6.2/app/cache
Deleting cache from config (70 files) ... ok
Deleting cache from Squished (4 files) ... ok
  • restart icinga
# /etc/init.d/icinga restart

Verify pnp4nagios and icinga-web with pnp4nagios

_images/pnp4nagios_env.png


  • remove install.php if test passes
# rm -f /usr/local/pnp4nagios-0.6.16/share/install.php
_images/pnp4nagios_overview.png
  • icinga-web with pnp4nagios(pop-up performance graph)
_images/icinga-web_pnp4nagios_01.png
  • performance graph in detail
_images/icinga-web_pnp4nagios_02.png


I'm going to introduce about configurations, especially init scripts.

Monday, April 30, 2012

Monitoring tool - install icinga and icinga-web

I've been using a monitoring tool, Nagios for several years. Though it is a classic monitoring tool and still in general use, icinga has several advantages compared with nagios, which shows here.

And the architecture of  Icinga is show in picture , here.

As Nagios is convertible with Icinga (it's not sure 100%), I am going to try migrate from Nagios-3.x to Icinga-1.x. The official says, here.

We try to be compatible with the configuration files of the current Nagios 3.x versions so there should be very little you have to do to “upgrade” from Nagios 3.x to Icinga 1.x. Assuming you’ve already installed Nagios from source code as described in the Nagios quickstart guide, you can install Icinga quite easily.

I am going to show how to install Icinga and migrate the current nagios-3.x configuration files in serial form.




  • Overview of System Structure
    • OS
    OS CentOS release 5.8 (Final)
    Kernel 2.6.18-274.el5 x86_64
    Language ja_JP.UTF-8
    • middleware
    role package
    WEB httpd-2.2.10
    DB mysql-5.5.21
    Scripting Language(for nagios-plugins) perl-5.14.2
    Scripting Language(for icinga-web) php-5.3.10
    icinga core icinga-1.6.1
    plugin nagios-plugins-1.4.15
    icinga frontend icinga-web-1.6.2
    icinga report icinga-reports-1.6.0
    reporting engine(for icinga-reports) jasperreports-server-cp-4.2.1
    graph(nagios addon) pnp4nagios-0.6.16
    • Directory structure of the middleware avobe
    /usr/local/
    |-- httpd -> /usr/local/httpd-2.2.22/
    |-- httpd-2.2.22/
    |-- icinga -> /usr/local/icinga-1.6.1/
    |-- icinga-1.6.1/
    |-- icinga-idoutils/
    |-- icinga-web -> /usr/local/icinga-web-1.6.2/
    |-- icinga-web-1.6.2/
    |-- jasperreports-server-cp-4.2.1/
    |-- mysql -> /usr/local/mysql-5.5.21/
    |-- mysql-5.5.21/
    |-- perl -> /usr/local/perl-5.14.2/
    |-- perl-5.14.2/
    |-- php -> /usr/local/php-5.3.10/
    `-- php-5.3.10/


 I am showing the process of installing icinga and icinga-web.

Icinga-core + NODUtils

  •  add user and group
# groupadd icinga && groupadd icinga-cmd
# useradd -G icinga-cmd,apache icinga 
  • install compiler, net-snmp and other dependet libaries
    $arch defines its architecture, i386 or x86_64. Here is x86_64(64bit OS)
# arch=$(uname -i)
# yum -y install gcc.${arch} glibc.${arch} glibc-common.${arch} gd.${arch} gd-devel
# yum -y install libjpeg.${arch} libjpeg-devel.${arch} libpng.${arch} libpng-devel.${arch} libdbi-devel.${arch}
# yum -y install net-snmp.${arch} net-snmp-devel.${arch} net-snmp-utils
  • install libdbi-drivers libdbi-dbd-mysql
# yum -y install libdbi-drivers.${arch} libdbi-dbd-mysql.${arch}
  • install icinga-core + idoutils
$ cd /usr/local/src/
$ curl "http://downloads.sourceforge.net/project/icinga/icinga/1.6.1/icinga-1.6.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Ficinga%2F&ts=1335254872&use_mirror=jaist" | tar zx
$ cd icinga-1.6.1
$ ./configure \
--prefix=/usr/local/icinga-1.6.1 \
--enable-nanosleep \
--enable-event-broker \
--enable-idoutils \
--enable-embedded-perl \
--enable-ssl \
--with-icinga-user=icinga \
--with-icinga-group=icinga \
--with-command-user=icinga \
--with-command-group=icinga-cmd \
--with-web-user=apache \
--with-web-group=apache \
--with-cgiurl=/icinga/cgi-bin \
--with-htmurl=/icinga \
--with-httpd-conf=/usr/local/httpd/conf/extra \
--with-checkresult-dir=/var/spool/checkresults \
--with-log-dir=/var/log/icinga \
--with-cgi-log-dir=/var/log/icinga \
--with-init-dir=/etc/init.d/ \
--with-lockfile=/var/run/icinga \
--with-ssl
$ make all
# make install-init
# make install-config
# make install-webconf
# make install-idoutils
# make install-commandmode
# make install
  • make a link with the prefix directory
# ln -s /usr/local/icinga /usr/local/icinga

  • setup configration, and enable the idomod event broker module
    The official shows the command below, which moves the sample configuration file, but ido2db.cfg and idmod.cfg are created.
# cd /usr/local/icinga/etc
# mv ido2db.cfg-sample ido2db.cfg
# mv idomod.cfg-sample idomod.cfg
# mv modules/idoutils.cfg-sample modules/idoutils.cfg
  • create mysql database
# mysqladmin -uroot create icinga
# mysql -uroot mysql -e "
GRANT USAGE ON *.* TO 'icinga'@'localhost'
   IDENTIFIED BY 'icinga'
   WITH MAX_QUERIES_PER_HOUR 0
   MAX_CONNECTIONS_PER_HOUR 0
   MAX_UPDATES_PER_HOUR 0;
GRANT ALL ON icinga.* TO 'icinga'@'localhost';
FLUSH PRIVILEGES;"
  • import database scheme
# mysql -uicinga -picinga icinga < /usr/local/src/icinga-1.6.1/module/idoutils/db/mysql/mysql.sql
  • install nagios-plugins
$ curl "http://jaist.dl.sourceforge.net/project/nagiosplug/nagiosplug/1.4.15/nagios-plugins-1.4.15.tar.gz" | tar xz
$ cd nagios-plugins-1.4.15
./configure \
--prefix=/usr/local/icinga \
--with-nagios-user=icinga \
--with-nagios-group=icinga \
--with-openssl \
--with-perl=/usr/local/perl/bin/perl
$ make
# make install

Icinga-web

  • install icinga-web
$ cd /usr/local/src/
$ curl "http://downloads.sourceforge.net/project/icinga/icinga-web/1.6.2/icinga-web-1.6.2.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Ficinga%2Ffiles%2Ficinga-web%2F1.6.2%2F&ts=1335497712&use_mirror=jaist" | tar zx
$ cd icinga-web-1.6.2
$ ./configure \
--prefix=/usr/local/icinga-web-1.6.2 \
--with-web-user=apache \
--with-web-group=apache \
--with-web-path=/icinga-web \
--with-web-apache-path=/usr/local/httpd/conf/extra \
--with-db-type=mysql \
--with-db-host=localhost \
--with-db-port=3306 \
--with-db-name=icinga_web \
--with-db-user=icinga_web \
--with-db-pass=icinga_web \
--with-db-socket=/tmp/mysql.sock \
--with-api-host=localhost \
--with-api-port=3306 \
--with-api-db-user=icinga \
--with-api-db-pass=icinga \
--with-api-db-name=icinga \
--with-api-db-prefix=icinga_ \
--with-api-cmd-file=/usr/local/icinga/var/rw/icinga.cmd \
--with-log-dir=/var/log/icinga \
--with-enable-sla
$ make
# make install
# make install-apache-config
# make install-javascript
  • test php modules' dependency
# make testdeps
/usr/local/bin/php bin/testdeps.php
Icinga-web dependencies (running 23 test)

PHP (core) tests
        1/23 Test php version >= 5.2.3 (REQUIRED) ... (version 5.3.10 >= 5.2.3) OK
        2/23 Test for PEAR (REQUIRED) ... OK

PHP extensions
        3/23 Test php5-xsl (REQUIRED) ... (xsl found v0.1) OK
        4/23 Test php5-ldap ... (ldap found) OK
        5/23 Test php5-pdo (REQUIRED) ... (PDO found v1.0.4dev) OK
        6/23 Test php5-dom (REQUIRED) ... (dom found v20031129) OK
        7/23 Test php5-session (REQUIRED) ... (session found) OK
        8/23 Test php5-spl (REQUIRED) ... (SPL found v0.2) OK
        9/23 Test php5-pcre (REQUIRED) ... (pcre found) OK
        10/23 Test php5-tokenizer (REQUIRED) ... (tokenizer found v0.1) OK
        11/23 Test php5-libxml (REQUIRED) ... (libxml found) OK
        12/23 Test php5-reflection (REQUIRED) ... (Reflection found v$Revision: 321634 $) OK
        13/23 Test php5-gettext (REQUIRED) ... (gettext found) OK

Optional pdo drivers
        14/23 Test php5-pdo-mysql ... (pdo_mysql found v1.0.2) OK
        15/23 Test php5-pdo-pgsql ... (Extension pdo_pgsql does not exist) FAIL

Optional php extension
        16/23 Test php5-soap ... (soap found) OK
        17/23 Test php5-xmlrpc ... (xmlrpc found v0.51) OK
        18/23 Test php5-iconv ... (iconv found) OK
        19/23 Test php5-gd ... (gd found) OK
        20/23 Test php5-ctype ... (ctype found) OK
        21/23 Test php5-json ... (json found v1.2.1) OK
        22/23 Test php5-hash ... (hash found v1.0) OK
        23/23 Test php.ini memory_limit ... (memory_limit='134217728') OK

All over result: PASS (required 12/12, optional 10/11, all 22/23, time 0.02s)

Exit (status=0)
  • create mysql database and user for icinga-web
# mysqladmin -uroot create icinga_web
# mysql -uroot mysql -e "
GRANT USAGE ON *.* TO 'icinga_web'@'localhost'
   IDENTIFIED BY 'icinga_web'
   WITH MAX_QUERIES_PER_HOUR 0
   MAX_CONNECTIONS_PER_HOUR 0
   MAX_UPDATES_PER_HOUR 0;
GRANT ALL ON icinga_web.* TO 'icinga_web'@'localhost';
FLUSH PRIVILEGES;"
  • initialize database
# make db-initialize
cd ./etc/ && ../bin/phing -f build.xml db-initialize
Buildfile: /usr/local/src/icinga-web-1.6.2/etc/build.xml
 [property] Loading /usr/local/src/icinga-web-1.6.2/etc/build.properties

icinga-web > db-initialize:

[phingcall] Calling Buildfile '/usr/local/src/icinga-web-1.6.2/etc/build.xml' with target 'db-create'
 [property] Loading /usr/local/src/icinga-web-1.6.2/etc/build.properties

icinga-web > prepare:

icinga-web > db-prepare:

Use the db-user specified in icinga-webs database.xml (Needs create Database/create Table rights)?(n,y)? y

icinga-web > db-create:

BUILD FINISHED

Total time: 3.4069 seconds
  • Test the syntax of icinga.cfg
    Though it is necessary to run Icinga daemon with the correct syntax of icinga.cfg and other related configuration files, such as nagios.cfg, commands.cfg, services.cfg, and so on, I am going to try show them later.
 # icinga -v /usr/local/icinga/etc/icinga.cfg 

Icinga 1.6.1
Copyright (c) 2009-2011 Icinga Development Team (http://www.icinga.org)
Copyright (c) 2009-2011 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 12-02-2011
License: GPL

Reading configuration data...
   Read main config file okay...
Processing object config directory '/usr/local/icinga/etc/modules'...
Processing object config file '/usr/local/icinga/etc/modules/idoutils.cfg'...
Processing object config directory '/usr/local/icinga/etc/objects'...
Processing object config file '/usr/local/icinga/etc/objects/commands.cfg'...
Processing object config file '/usr/local/icinga/etc/objects/contacts.cfg'...
Processing object config file '/usr/local/icinga/etc/objects/timeperiods.cfg'...
Processing object config file '/usr/local/icinga/etc/objects/nagios_perfparse.cfg'...
Processing object config file '/usr/local/icinga/etc/objects/templates.cfg'...
Processing object config directory '/usr/local/icinga/etc/templates'...
Processing object config file '/usr/local/icinga/etc/templates/hosts.cfg'...
Processing object config file '/usr/local/icinga/etc/templates/services_mon.cfg'...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking services...
        Checked 32 services.
Checking hosts...
        Checked 1 hosts.
Checking host groups...
        Checked 1 host groups.
Checking service groups...
        Checked 0 service groups.
Checking contacts...
        Checked 1 contacts.
Checking contact groups...
        Checked 1 contact groups.
Checking service escalations...
        Checked 0 service escalations.
Checking service dependencies...
        Checked 0 service dependencies.
Checking host escalations...
        Checked 0 host escalations.
Checking host dependencies...
        Checked 0 host dependencies.
Checking commands...
        Checked 47 commands.
Checking time periods...
        Checked 3 time periods.
Checking modules...
        Checked 1 modules.
Checking for circular paths between hosts...
Checking for circular host and service dependencies...
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
  • start icinga daemon
# /etc/init.d/icinga start
Running configuration check...OK
Starting icinga: Starting icinga done.
  • restart apache daemon
# /etc/init.d/httpd restart
  • Top page after logging in

  • A table of items’ status
Next will be how to install icinga-reports, pnp4nagios or about the details of configuration and init script.

iJAWS@Doorkeeper