Monitoring tool - monitor icinga/nagios with monit

As introducing of how to install monit before, I am explaining how to monitor icinga/nagios with monit. Monit has several kinds of testing and defines them here. I adopt PPID testing which tests the process parent process identification number (ppid) of a process for changes to check icinga daemon.

The configurations for service entry statement are released on my github.

Configuration

• setup pidfile of icinga.cfg(nagios.cfg)
  Though the directive says "lock_file", it actually outputs the process id number to the file. Nagios official says, here.

  "This option specifies the location of the lock file that Nagios should create when it runs as a daemon (when started with the -d command line argument). This file contains the process id (PID) number of the running Nagios process."

# grep '^lock_file' icinga.cfg 
lock_file=/var/run/icinga.pid

# /etc/init.d/icinga reload

• setup service entry statement of icinga

# cat > /etc/monit.d/icinga.conf >> EOF
check process icinga
      with pidfile "/var/run/icinga.pid"
      start program = "/etc/init.d/icinga start"
      stop program = "/etc/init.d/icinga stop"
      if 3 restarts within 3 cycles then alert

EOF

Start up

• begin monitoring

# monit monitor icinga
# monit start icinga

• see the summary

# monit summary | grep 'icinga'
Process 'icinga'                    Running

• see the monit log file

# tail -f /var/log/monit/monit.log
[JST May 13 14:35:48] info     : 'icinga' monitor on user request
[JST May 13 14:35:48] info     : monit daemon at 13661 awakened
[JST May 13 14:35:48] info     : Awakened by User defined signal 1
[JST May 13 14:35:48] info     : 'icinga' monitor action done
[JST May 13 14:37:07] error    : monit: invalid argument -- staus  (-h will show valid arguments)
[JST May 13 14:37:39] info     : 'icinga' start on user request
[JST May 13 14:37:39] info     : monit daemon at 13661 awakened
[JST May 13 14:37:39] info     : Awakened by User defined signal 1
[JST May 13 14:37:39] info     : 'icinga' start action done

Verification 

• verify icinga daemon begins if  its process is stopped 

# /etc/init.d/icinga status
icinga (pid  31107) is running...

# kill `pgrep icinga`

• see the log file that monit begins icinga

# cat /var/log/monit/monit.log
[JST May 13 14:37:39] info     : 'icinga' start on user request
[JST May 13 14:37:39] info     : monit daemon at 13661 awakened
[JST May 13 14:37:39] info     : Awakened by User defined signal 1
[JST May 13 14:37:39] info     : 'icinga' start action done
[JST May 13 14:45:40] error    : 'icinga' process is not running
[JST May 13 14:45:40] info     : 'icinga' trying to restart
[JST May 13 14:45:40] info     : 'icinga' start: /etc/init.d/icinga

• check icinga is running.

# /etc/init.d/icinga status
icinga (pid  21093) is running...

Configuration examples(ido2db, npcd)

• setup pidfile of ido2db.cfg (ndo2db)

# grep '^lock_file' ido2db.cfg 
lock_file=/var/run/ido2db.pid

• setup service entry statement of ido2db

# cat > /etc/monit.d/ido2db.monit << EOF
check process ido2db
      with pidfile "/var/run/ido2db.pid"
      start program = "/etc/init.d/ido2db start"
      stop program = "/etc/init.d/ido2db stop"
      if 3 restarts within 3 cycles then alert

EOF

• begin monitoring

# monit monitor ido2db
# monit start ido2db

• setup pidfile of npcd.cfg (pnp4nagios)

# grep '^pid_file' npcd.cfg 
pid_file=/var/run/npcd.pid

• setup service entry statement of npcd

# cat > /etc/monit.d/npcd.monit << EOF
check process npcd
      with pidfile "/var/run/npcd.pid"
      start program = "/etc/init.d/npcd start"
      stop program = "/etc/init.d/npcd stop"
      if 3 restarts within 3 cycles then alert

EOF

• begin monitoring

# monit monitor npcd
# monit start npcd