As discussed in my previous article on server monitoring, if you care about the uptime of your server, you need to have automated monitoring in place.

monit is a great open-source server monitoring tool, which you install on your Linux server. It can:

• Constantly monitor any service or system status
• Restart services when needed
• Send alerts
• Provide a monitoring console

Note: These instructions reflect the Rails Machine system configuration, and you’ll need to make adjustments for other Linux variants and directory arrangements. Thanks to Bradley Taylor for providing the instructions from which I started.

Install monit

First, install monit:

wget http://dag.wieers.com/rpm/packages/monit/monit-4.9-1.el4.rf.i386.rpm
sudo rpm -i monit-4.9-1.el4.rf.i386.rpm

(These file names reflect the current version as of this writing; you’ll need to update them as new versions are released.)

Configure monit

monit is highly configurable, and it takes a little work to get it started. First, make a backup of the original config file:

sudo cp /etc/monit.conf /etc/monit.conf.orig

Now edit /etc/monit.conf and make the following changes:

• uncomment the line “set daemon 120”, so monit will run every two minutes
• uncomment the line that starts “set alert” and put in the email address to which you want alerts sent
• uncomment the line “include /etc/monit.d/*.conf”

This last change causes monit to include all .conf files from /etc/monit.d as part of the configuration. This allows you to use a separate config file for each service, which makes it easier to maintain the configuration as things change.

Configure Services to be Monitored

Now create files in /etc/monit.d to configure each service to be monitored. The file names don’t matter, since all files in this directory that end in ”.conf” will be included as part of the confguration.

Monitoring Memory, Disk, and CPU Usage

The following configuration, stored in /etc/monit.d/system.conf, monitors the basic health of the server:

check system redtail.mzslater.railsmachina.com

  if loadavg (1min) > 4 then alert

  if loadavg (5min) > 2 then alert

  if memory usage > 85% then alert

check device rootfs with path /dev/sda1

  if space usage > 85% then alert

Monitoring Apache

This is my Apache monitoring configuration, which is stored in /etc/monit.d/httpd.conf:

check process httpd with pidfile /var/log/httpd/httpd.pid
   group apache
   start program = "/etc/init.d/httpd start" 
   stop  program = "/etc/init.d/httpd stop" 
   if failed host 127.0.0.1 port 80
        protocol HTTP request /monit/token then restart
   if 5 restarts within 5 cycles then timeout

Monitoring MySQL

The MySQL configuration goes in the file /etc/monit.d/mysqld.conf:

check process mysql with pidfile /var/run/mysqld/mysqld.pid
  group database
  start program = "/etc/init.d/mysqld start" 
  stop program = "/etc/init.d/mysqld stop" 
  if failed host 127.0.0.1 port 3306 protocol mysql then restart
  if 5 restarts within 5 cycles then timeout

Monitoring Mongrel Cluster

If you’re running a Rails application, you’ll also want to monitor whatever you’re using as the Rails server. In my case, it is two Mongrel instances. Here’s my basic configuration, stored in /etc/monit.d/mongrel.conf:

check process mongrel-8000 with pidfile /var/www/apps/[appname]/shared/log/mongrel.8000.pid
     start program = "/usr/bin/mongrel_rails cluster::start -C /etc/mongrel_cluster/[appname].conf" 
     stop program  = "/usr/bin/mongrel_rails cluster::stop -C /etc/mongrel_cluster/[appname].conf" 
     # restart if memory usage too high
     if totalmem is greater than 60.0 MB for 5 cycles then restart   
     # send an email to admin if cpu load too high
     if cpu is greater than 50% for 2 cycles then alert                  
     # restart if hung process
     if cpu is greater than 80% for 3 cycles then restart
     if loadavg(5min) greater than 10 for 8 cycles then restart
     # if problems repeat, call the sys-admin    
     if 3 restarts within 5 cycles then timeout                                         
     # check for response
     if failed port 8000 protocol http                                                         
        with timeout 10 seconds
        then restart
     group mongrel

Replace [appname] with the name of your application, and 8000 with the port number that your first mongrel instance is using. The paths may also be different for your installation.

Repeat this configuration code for each mongrel instance, updating the port number for each.

Configure Apache to Not Log monit Requests

Since monit is going to be hitting your web site regularly, you don’t want it cluttering up your logs and distorting your statistics. To stop Apache from logging monit’s requests, add the following lines to the end of your httpd.conf file (in my case, this file is in /etc/httpd/conf).

SetEnvIf Request_URI ”^\/monit\/token$” dontlog
CustomLog logs/access.log common env=!dontlog

You may need to modify the second line to reflect where you’re putting your logs. And, of course, if you’re using a server other than Apache, you’ll need to use that server’s configuration syntax to disable logging of requests to /monit/token.

Since we’ve changed its configuration, restart Apache:

sudo /sbin/service httpd reload

Create a test page

Create an empty file for Monit to request from Apache to test that the web server is alive:

mkdir /var/www/html/monit
touch /var/www/html/monit/token

Start Monit

Finally, enable and start Monit:

sudo /sbin/chkconfig monit on
sudo /sbin/service monit start