noizZze

Scheduling Sidekiq (Resque) Tasks From Console

In my projects, traditionally, I use cron to schedule periodical maintenance tasks and then do all the heavy lifting in the background. Cron would start a Rake task that would, in turn, schedule a job. Then Sidekiq (or Resque, previously) pick it and take care.

Recently I became aware of increased CPU and memory usage on the background workers server. Quick research showed that Rake tasks started taking a great deal of resources. While we had just several that started once an hour or even less frequently, everything went smoothly. As we’ve got some minutely tasks, things slowly went out of hand.

It’s obvious that using a Rake task for scheduling a job is an overkill. All the task does is placing a record in the Redis database with job details, so why spending 30 seconds loading whole Rails stack? Unfortunately, I couldn’t find any mention of starting tasks from the shell, so I went into the redis-cli tool and checked the queue for what exactly is being pushed in there. That appeared to be a simply crafted bit of JSON.

Here’s what I run to push a job (Pulse class) with no arguments to a “default” queue:

redis-cli rpush queue:default "{\"retry\":true,\"queue\":\"default\",\"class\":\"Pulse\",\"args\":[],\"jid\":\"db:pulse-`date +%s`\"}"

The only part deserving a bit of details is the “jid” parameter. I use a simple notion saying which node the task comes from (db), what is it (pulse) and a simple unix timestamp to both ensure it’s unique and know when it happened. As simple as that.

You can imagine how fast is it compared to the Rake business.

Comments