Redis and Resque Customizations

For one of the projects I’m working on for my clients we need a robust and easy to use queue. Redis and Resque is a great combination that fits us well except that we can’t:

  • Vary the interval between polling queues. When you know how much exactly you have to sleep before the next job can be picked for execution, there’s no need to waste precious time waiting longer than that.
  • Skip queues under certain conditions. If you know you need to pause a queue (throttling, for example), you couldn’t do it without hacks. Now you can.
  • Expand queue name patterns into the real queue names. Sometimes we don’t know which queues we’ll use (dynamic names), but we know the pattern (say, ‘mt_*’). You can write your custom queue lister that will convert any notation you want into the list of queues at run time (before the polling begins).
  • Specify the target queue when scheduling a job. Say, I have a SendMessage job and I want to prioritize the delivery for certain┬árecipients.

To address these problems I forked the Resque repository. The results you can find in my fork with nice documentation and full test suite.

I’m also thinking of adding queue meta-data functionality to let workers / jobs assign bits of information to the queues for later reference. In my case, I want to know when the last job was processed in this queue to be able to skip over the queues crossing the throttling threshold.