Monitoring services, CPU, disk and memory usage has become a common practice for IT companies big and small, but what often overseen are Cron jobs. (If you are unfamiliar with Cron, it’s the standard *nix way of running periodical tasks.) In this short post I show one simple way just to get you going.

In one of the companies I work with at the moment, they have several web apps with many Cron jobs. Jobs do billing, notifying, report processing and many other tasks. It happened several times that critical tasks stopped running under some circumstances, but there was no way of knowing that until it’s too late and someone’s reporting a problem.

The simplest solution I could come up with was touching a certain file (changing that file’s access timestamp to current time) after the successful task execution. Every Cron job would have its own file. Then the last bit would be Monit looking at these files and sending alerts when any of them weren’t touched longer than the period of corresponding tasks.

Here’s a sample Rake task:

1
2
3
4
5
6
7
8

namespace :app do
  task :billing => :environment do
    # Task logic ...

    # Now touch the file to indicate we finished the job successfully
    FileUtils.touch Rails.root.join('log', 'job_billing')
  end
end

After the billing task is complete, we touch the file log/job_billing. Given that we run this task daily, we can configure Monit to look at this file to see if it’s touched with these two lines:

1
2

check file app_job_billing with path /app_path/log/job_billing
    if timestamp > 25 hours then alert

I give it extra hour just in case, but you can choose the period yourself.

One more tip I’d like to give is keeping the Monit configuration in the config/monit.rc.erb with the rest of the sources, and generating the actual monit.rc that to include into /etc/monitrc on the server with a Capistrano task:

1
2
3
4
5
6
7

namespace :deploy do
  desc "Render monit configuration"
  task :monit, :roles => :app do
    c = ERB.new(File.read(File.expand_path("../monit.rc.erb", __FILE__)))
    put c.result(binding), "#{shared_path}/config/monit.rc"
  end
end

The tricky part here is to know the location of touched files that are inside the project directory hierarchy (yes, you can put them under /tmp, but I prefer to keep project-related stuff with the project). That’s why I’m using ERB templates and render them at the moment when the shared directory path on the remote server is known. Here’s the template:

1
2

check file app_job_billing with path <%= shared_path %>/log/job_billing
    if timestamp > 25 hours then alert

Hopefully this’ll make your products more stable!