noizZze

Monit Configuration for Rails Stack

It’s more like a self-note, but also an answer to a question what I like to monitor on the servers where we run Rails apps and how. Thought that it could save some time to anyone. Let me know if you have any questions. I like to create the directory for Monit configuration bits and keep different services separate. It easier to update and reuse this way.

First comes some trivial stuff. You’ll notice that I start and stop services through their init.d binds, and checks are fairly typical.

Apache:

check process apache
  with pidfile "/var/run/httpd.pid"
  start program = "/etc/init.d/httpd start"
  stop program = "/etc/init.d/httpd stop"
  if children > 35 for 3 cycles then restart

Redis:

check process redis
  with pidfile /var/run/redis.pid
  start program = "/etc/init.d/redis start"
  stop program = "/etc/init.d/redis stop"
  if failed host 127.0.0.1 port 6379 then restart
  if 5 restarts within 5 cycles then timeout

Mongo:

check process mongodb
  with pidfile /var/run/mongodb.pid
  start program = "/etc/init.d/mongodb start"
  stop program = "/etc/init.d/mongodb stop"
  if failed host 127.0.0.1 port 27017 then restart
  if 5 restarts within 5 cycles then timeout

Nginx:

check process nginx
  with pidfile /var/run/nginx.pid
  start program = "/etc/init.d/nginx start"
  stop program = "/etc/init.d/nginx stop"
  if failed host 127.0.0.1 port 80 then restart
  if 5 restarts within 5 cycles then timeout

Resque

Finally something interesting. To start workers I’m using the daemon-spawn gem. It’s a great wrapper that lets you write config in Ruby and run any class as a daemon with all bells and whistles like PID files, log files and arbitrary number of instances. Before Monit, I usually started several instances at once. This worked just great except it left only one PID behind and so monitoring and managing a single worker was a problem.

After a bit of toying with spawner and Monit I came up with the solution that works perfectly well for us these days. I generate the Monit config with the given number of workers of each job type in the format:

check process resque-GROUP-NUM with pidfile /home/deploy/project/current/tmp/pids/GROUP_worker_NUM.pid
  start program = "/bin/sh -c 'cd /home/deploy/project/current; N=1 RAILS_ENV=production HOME=/home/deploy daemons/GROUP_worker start'" as uid deploy and gid deploy
  stop program = "/bin/sh -c 'cd /home/deploy/project/current; N=1 RAILS_ENV=production HOME=/home/deploy daemons/GROUP_worker stop'"
  group sl

GROUP is the name of the job type (can be anything you like), and NUM is the counter (for the queue with 3 workers counts from 1 to 3, easy).

The final bit is the “RAILS_ROOT/daemons/GROUP_worker” script that manages workers of certain type:

#!/usr/bin/ruby

ENV['RAILS_ENV'] ||= 'development'
require File.expand_path('../../config/boot',  __FILE__)
require Rails.root.join("config", "environment")

class TaskWorkerDaemon < DaemonSpawn::Base
  def start(args)
    @worker = Resque::Worker.new('mo')
    @worker.verbose = 1
    @worker.work(5)
  end

  def stop
    @worker.try(:shutdown)
  end
end

n = ENV['N']

TaskWorkerDaemon.spawn!({
  :processes   => 1,
  :working_dir => RAILS_ROOT,
  :log_file    => File.join(RAILS_ROOT, "log", "task_worker_#{n}.log"),
  :pid_file    => File.join(RAILS_ROOT, "tmp", "pids", "task_worker_#{n}.pid"),
  :sync_log    => true,
  :singleton   => true })

As you can see, we have a worker class definition that we use to spawn just one process with certain PID file and log. Note that path to this PID file should match the on you feed to Monit. Otherwise there won’t be a connection between the two and Monit will keep trying to start the service that’s running until it gives up and stops monitoring.

Hope it helps someone. Drop me a message if you have a better config for anything.