wwwhizz' blog Jan Kazemier & his music, photography, nerd-projects, developing, electronics and gadgets

su: Bad item passed to pam_*_item()

/etc/default/locale

LANG=en_US.UTF-8

check process apache2 with pidfile /var/run/apache2.pid
alert myemail@adress.ext only on { timeout, nonexist }

check system myserver
alert foo@bar but not on { instance }

alert myemail@adress.ext on { action
checksum
content
data
exec
gid
icmp
invalid
fsflags
nonexist
permission
pid
ppid
size
timeout
timestamp }

Monitor your system, processes, daemons and remote hosts on Debian linux (lenny) with monit.

Since a while I am monitoring a few remote hosts, as well as my own server (serving you this page right now) with monit.

Their site says:

"Monit is a free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations."

Time to get started!

The installation is very easy with aptitude:

# aptitude install monit

If for some reason you have to compile it yourself:

$ tar zxvf monit-x.y.z.tar.gz (were x.y.z denotes version numbers)
$ cd monit-x.y.z
$ ./configure (use ./configure --help to view available options)
$ make && make install

Now it's time to configure. Monit is highly customizable

The monit configuration file is found at /etc/monit/monitrc

The configuration below can be found at
http://www.t44.nl/wordpress/data/monitrc

Here is an example you can edit:

##### Global options #####
## Logging options
# Poll every minute
set daemon  60                          # polling time in seconds
set logfile syslog facility log_daemon  # Set syslog logging.

## Mail options
# Here you can setup your own mailserver
set mailserver smtp.yourisp.ext,        # primary mailserver
               localhost                # fallback relay

If you would like to use GMail (which requires TLS/SSL) use something like this instead:

# set mailserver smtp.gmail.com port 587 username "myemail@gmail.com" password "password" using tlsv1 with timeout 30 seconds

The eventqueue holds events if for some reason mails cannot be sent.

set eventqueue
     basedir /var/monit # set the base directory where events will be stored
     slots 100          # optionaly limit the queue size

You can define your mail-notification format. Use man monit for all options.

set mail-format {
## Just an example, feel free to edit
    from: monit@yourhost.com
    subject: (monit) $SERVICE $EVENT at $DATE
    message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
    Yours sincerely,
    monit
    Your system admin
}

All alert mail will be sent to below mail address.

# Set a default mail from-address for all alert messages emitted by monit.
set alert you@something.ext # fill in your e-mailaddress here

Let's setup the webinterface

# Web-server options
  set httpd port 80          # set httpd port for the web-interface,
                                 # default is 80, 443 is normally used for https
  allow my.home.ip.address     # Only these addresses can connect to the webinterface
  allow localhost                # All hosts can connect when omitted
  allow username:password      # set your users/passwds here
  allow foo:bar                  # Multiple users are allowed

If you wish (like me) to use SSL connections:

# Web-server options
  set httpd port 443           # set httpd port for the web-interface,
                                 # default is 80, 443 is normally used for https
  ssl enable                   # if you wish to enable ssl (i.e. https) connections
  PEMFILE /var/certs/monit.pem   # (only for SSL)
  allow my.home.ip.address     # Only these addresses can connect to the webinterface
  allow localhost                # All hosts can connect when omitted
  allow username:password      # set your users/passwds here
  allow foo:bar                  # Multiple users are allowed

This requires a certificate, and some more than basic knowledge. Much of it is very well explained in the Monit Wiki

Let's continue setting up services. First of all we check wheter our own system is running nicely. This is pretty straightforward.

##### Services #####
## System
  check system localhost
     if loadavg (5min) > 1 then alert
     if memory usage > 75% then alert
     if cpu usage (user) > 70% then alert
     if cpu usage (system) > 30% then alert
     if cpu usage (wait) > 20% then alert

Now we will monitor a few processes. The pid-files mostly are in /var/run/ (at least in Debian).

## Processes
# apache2
check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop  program = "/etc/init.d/apache2 stop"
   if failed host 127.0.0.1 port 80 protocol http and request "/monit/token" with timeout 15 seconds then restart
   if cpu > 60% for 2 cycles then alert
   if cpu > 90% for 5 cycles then restart
   if totalmem > 200.0 MB for 5 cycles then restart
   if children > 250 then restart
   if failed host 127.0.0.1 port 50443 type tcpssl protocol http and request "/monit/token" with timeout 15 seconds then restart
   if 5 restarts within 5 cycles then timeout
   depends on apache_bin
   depends on apache_rc

 check file apache_bin with path /usr/sbin/apache2
   group www
   if failed checksum then unmonitor
   if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor

 check file apache_rc with path /etc/init.d/apache2
   group www
   if failed checksum then unmonitor
   if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor

Over to MySQL

# mysql
check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host localhost port 3306 protocol mysql then restart
   if 5 restarts within 5 cycles then timeout
   depends on mysql_bin
   depends on mysql_rc

 check file mysql_bin with path /usr/bin/mysql
   group database
   if failed checksum then unmonitor
   if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor

 check file mysql_rc with path /etc/init.d/mysql
   group database
   if failed checksum then unmonitor
   if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor

And syslogd:

# syslogd
check process syslogd with pidfile /var/run/syslogd.pid
   start program = "/etc/init.d/sysklogd start"
   stop program = "/etc/init.d/sysklogd stop"
   if 5 restarts within 5 cycles then timeout

 check file syslogd_file with path /var/log/syslog
   if timestamp > 65 minutes then alert

sshd and Samba:

# sshd
 check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/ssh start"
   stop program  "/etc/init.d/ssh stop"
   if failed host localhost port 50022 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

# Samba
 check process smbd with pidfile /var/run/samba/smbd.pid
   group samba
   start program = "/etc/init.d/samba start"
   stop  program = "/etc/init.d/samba stop"
   if failed host localhost port 139 type TCP  then restart
   if 5 restarts within 5 cycles then timeout
   depends on smbd_bin

 check file smbd_bin with path /etc/init.d/samba
   group samba
   if failed checksum then unmonitor
   if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor

Finally we also check wheter some other hosts are still up and running.

## Check other hosts
  check host google.com with address google.com
    if failed icmp type echo count 3 with timeout 3 seconds then alert
    if failed port 80 protocol http then alert
  check host t44.nl with address t44.nl
    if failed icmp type echo count 3 with timeout 3 seconds then alert
    if failed port 80 protocol http then alert

It is also possible to include some other configuration files with the include keyword:

##############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#  For instance:
include /etc/monit.d/*
#
#

After modifying the configuration file you should check for the syntax to make sure they are correct. To do this run:

# monit -t

Now you can run monit standalone:

# monit

Now you should be able to check monit in your webbrowser, and probably you'll get a few alert mails If for some reason you cannot access the webbrowser, you can throw out all lines that say

allow *hostname*

This will make monit accept all IP-adresses and hosts. Also check that your host correctly has port 80/443 (or whatever you have setup) forwarded, and that your firewall and ip-tables are set up correctly.

In order to use the Debian init.d startup script you must enable the service by changing the file /etc/default/monit.

# You must set this variable to for monit to start
startup=1

# To change the intervals which monit should run uncomment
# and change this variable.
CHECK_INTERVALS=30

Now you can start monit by running /etc/init.d/monit start. Also it will automatically start on reboot.