Securing your sleep when running Mongrel
Mongrel, which, along with Apache 2.2 & load balancer & mongrel cluster, I run for hosting my Rails site - is a very nice piece of software. But there is a problem. It crashes.
I found about it the hard way - by a phone call at 0:30 in the morning, saying "the server seems to be down". "Doh", I though - and got out of my comfy bad.. Got to my computer, logged in to the server, restarted Mongrel cluster and got back to sleep (you can always check the logs later in the morning, right? :)
But well it turned out that the night was still not over just yet. Seems that I have a some kind of mental connection to my server, as I woke up at around 4am just to "check things out" because somehow couldn't sleep - just to find out that these things were worth checking - server was giving me Error 503, and not a single Mongrel instances was to be found alive. This time I checked logs - Apache is fine, just "hey, I can't find any load balancer workers, huh?", and Mongrel's logs were clean as well - they just happily stopped logging at around 3:30am, without a trace of error of stuff like that. Looks like Mongrel just decided to take a break :)
Its pretty understandable that Mongrel at its barely 1.0 version (it was at version 0.3-something just a month ago) is not the most stable thing in the world. But I needed a good sleep at night, and with server crashing unexpectedly and without any errors, the only solution was to install some monitoring software to bring the thing to life when I goes down.
That's how I found about Monit. "Monit makes Mongrel play nice!" blog entry had some very nice info about setting up the monitoring software.
You install it up like this:
wget http://www.tildeslash.com/monit/dist/monit-4.8.1.tar.gz
tar zxvf monit*
cd monit*
./configure
make && make install
And once it is done, you have to create a configuration file for Monit to use. I have created one in ~/.monitrc. And it looked like this.
set daemon 60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@localhost }
set alert example@example.comcheck process httpd with pidfile /usr/local/apache2/logs/httpd.pid
group www-data
start program "/usr/local/apache2/bin/apachectl start"
stop program "/usr/local/apache2/bin/apachectl stop"
if failed host localhost port 80 protocol http
and request "/" then alert
if cpu is greater than 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then alert
if 3 restarts within 5 cycles then timeoutcheck process mongrel8100 with pidfile /webhome/mysite/shared/log/mongrel.8100.pid
group mongrel
start program = "/usr/local/bin/ruby /usr/local/bin/mongrel_rails start -d -e production -p 8100 -a 127.0.0.1 -P log/mongrel.8100.pid -c /webhome/mysite/current"
stop program = "/usr/local/bin/mongrel_rails stop -P /webhome/mysite/shared/log/mongrel.8100.pid"
if failed host 127.0.0.1 port 8000 protocol http
and request "/" then alert
if totalmem > 100 Mb then restart
if cpu is greater than 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if loadavg(5min) greater than 10 for 8 cycles then restart
if 3 restarts within 5 cycles then timeout
The last part "check process mongrel8100 with pidfile…" had to be repeated for each mongrel instance launched inside the mongrel cluster. (just changin the pid numbers, and paths in case of instances working for other sites on the server).
Once you are done, you can launch the monit like this:
monit -c /path/to/monitrc
It is not neccessary to specify the -c parameter in case you placed the monitrc file in a place monit knows about (~/.monitrc, /etc/monitrc, /usr/local/etc/monitrc, ./monitrc). It would be a good idea to launch monit at server startup of course :)
Now.. it something happens to a mongrel instance…
..monit will automatically relaunch the service for you! :)
Piece of mind and good night sleep guaranteed.
You can also notice I have added monitoring for Apache. Just in case you know.. Although Apache didn't crash a single time on me yet, for.. how many years?..