noizZze

Using Sidekiq With Elastic Beanstalk

I’m a big fan of Sidekiq, but recently, one of my clients asked for hosting their new application on Amazon’s Elastic Beanstalk which imposed certain limitations, that I had to overcome to get my normal setup working. In this post I want to describe some of them.

I usually either use Sidekiq on the same node, or place it on a dedicated one to put demanding workers away from the main web app. It requires Redis to maintain the queue of tasks and the copy of application to be able to execute them.

Amazon’s Elastic Beanstalk is the framework with load balancing and autoscaling that lets you focus on your application, not infrastructure. The copy of your Rails application (in this case) is unziped and configured on each instance with Nginx and Passenger. If you need database, they provide RDS that runs MySQL (in this case).

It appears that the setup with a dedicated node for Sidekiq and one with Redis isn’t quite possible with current state of Beanstalk. There’s no distinction between nodes – every one of them is just like the sibling. At the same time, there’s a notion of a “leader”, being the first node handled during the deployment, but we can’t use it to identify a certain node reliably afterwards. It can be used for migrations or other one-off tasks during the deployment only, and can change from deployment to deployment. On top of this, the auto-scaler is free to terminate any instance during downscaling, and I read it that it leans towards older instances more often than not.

Here’s what I finally came up with:

  • use RedisToGo as an external Redis database. They host it on EC2 in the same zone and so the latency is super-small.
  • deploy workers on each web node. Even though it’s an unusual setup, it makes sence. As the load grows, the number of tasks to run is likely to increase as well, and so scaling workers makes sense.

To (re)start workers on the nodes, I placed this .ebextensions/sidekiq.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq":
    mode: "000777"
    content: |
      cd /var/app/current

      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -TERM `cat /var/app/support/pids/sidekiq.pid`
        rm -rf /var/app/support/pids/sidekiq.pid
      fi

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      bundle exec sidekiq \
        -e production \
        -P /var/app/support/pids/sidekiq.pid \
        -C /var/app/current/config/sidekiq.yml \
        -L /var/app/support/logs/sidekiq.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq":
    mode: "000777"
    content: |
      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -USR1 `cat /var/app/support/pids/sidekiq.pid`
      fi

Basically, after unziping (pre-init task 1) and setting up environment (pre-init task 2), we tell Sidekiq not to accept any more jobs. Then new app version is deployed, and finally (post-init task 50), we restart workers by first, stopping them, giving time to terminate, and starting over again.

Figuring this out took about two days of dirty time, and so hopefully, this will be of any help to others.

Update - Feb 17, 2014

I was contacted by guys from Redis Cloud who generously set me up with one of their premium plans for some testing of their advanced features. What they offer is something that had to be there for quite some time yet, most notably:

  • Amazon AWS nodes – gents keep their nodes on Amazon EC2 infrastructure. You can choose the zone when creating subscriptions to keep latency with your other EC2 nodes close to nil.

  • Amazon Security Groups auth – loving this. If you are on Amazon AWS for the rest of your projects – there’s no need to use passwords for auth. Just use Security Groups to let your work nodes talk to Redis.

  • Seamless failovers – you keep your endpoint, and they almost instantly set you up with another database in case of the crash or node outage. It’s hard to test this kind of feature, but I can say that during a week of testing on a real system with several connections, there were no outages reported.

  • Instant Backups – they do that, and that’s useful.

  • Increased number of connections

  • Low pricing – that’s something I loved. For more resources we now (yes, I’m using them for a new project) pay much less.

From what I’m not using yet there are auto-scaling and replication. So far I’m enjoying the services and hope it keeps going this smoothly. Give it a try.

Scheduling Sidekiq (Resque) Tasks From Console

In my projects, traditionally, I use cron to schedule periodical maintenance tasks and then do all the heavy lifting in the background. Cron would start a Rake task that would, in turn, schedule a job. Then Sidekiq (or Resque, previously) pick it and take care.

Recently I became aware of increased CPU and memory usage on the background workers server. Quick research showed that Rake tasks started taking a great deal of resources. While we had just several that started once an hour or even less frequently, everything went smoothly. As we’ve got some minutely tasks, things slowly went out of hand.

It’s obvious that using a Rake task for scheduling a job is an overkill. All the task does is placing a record in the Redis database with job details, so why spending 30 seconds loading whole Rails stack? Unfortunately, I couldn’t find any mention of starting tasks from the shell, so I went into the redis-cli tool and checked the queue for what exactly is being pushed in there. That appeared to be a simply crafted bit of JSON.

Here’s what I run to push a job (Pulse class) with no arguments to a “default” queue:

redis-cli rpush queue:default "{\"retry\":true,\"queue\":\"default\",\"class\":\"Pulse\",\"args\":[],\"jid\":\"db:pulse-`date +%s`\"}"

The only part deserving a bit of details is the “jid” parameter. I use a simple notion saying which node the task comes from (db), what is it (pulse) and a simple unix timestamp to both ensure it’s unique and know when it happened. As simple as that.

You can imagine how fast is it compared to the Rake business.

Catching Up

Oh, it’s been more than half a year since the last post! I figured, if there’s someone still following my blog, they deserve this post.

So what’s up with us:

  • Max quickly becomes a fully formed man with the character, good dictionary and is already capable of building 3 or 4-word sentences.

  • a couple of months ago we finally found a place of our own – a gorgeous 3 bedroom apartment with a spacious living room, fully equipped kitchen and great location in relation to our families

  • I’m working on three exciting projects for clients involving video and photo processing, native clients for Android and iOS, paid subscriptions with PayPal, and a ton of other curious technological advances

  • … as well as taking two online Stanford classes on building a startup and finances

  • Kate’s hoping to resurrect her hand made bags business one day

  • We finally relaxed a bit and stopped saving money for anything big. It cut us some slack and gave well-deserved peace of mind

  • Another winter season is coming and I’m anticipating lots of snowboarding, travelling and fun guiding Max through his second winter

  • A couple of days ago I ran across the announcement of Ableton Live 9 and Push that are due this coming winter. Immediately started dreaming of resuming my musical workings. I’m mulling over many enjoyable ideas to put on musical canvas eventually, so more stuff to come soon

That’s about all I have for you, fellas. I’m hoping to write more often, but hey, I hope it each time.

Until next time!

Speed-up Testing When Using Sorcery

I’m using RSpec in most of my recent projects. In fact, whatever new I start gets RSpec installed by default. Recently I began using Sorcery gem for authentication. It’s a small and non-intrusive solution that I find very well-fitted for how I do things. If you are using Sorcery, you may be glad to find out that it’s possible to make your whole functional testing suite almost twice as fast.

Prime Numbers Challenge

Yesterday, while I was reading a book on Erlang, I’ve got this idea to devise and implement prime numbers finding algorithm without looking up anything. So I did. But then figured my friends may also want to get their feet wet and so I tweeted about the “challenge” and got 3 submissions (2 in Ruby and 1 in Erlang). Later, I discovered that a couple more would have wanted to try but never saw my tweet. Well, it’s a shame.

Erlang Hints

I’m reading excellent “Programming Erlang” by Joe Armstrong and find some new stuff I haven’t seen in “Erlang programming” by Francesco Cesarini. This is largely a self-note post, so take it with a grain of salt.

Lists

  • If your code prepends elements effectively reversing the order of the original list, you need to call lists:reverse(...) as opposed to concatenating lists (L ++ [ X ]). The reverse call is way more efficient and even though it’s defined in lists module, when VM sees the call, it invokes an internal version.

Exception handling

  • The value of the after block is lost
  • The after block can be omitted
  • If you don’t need any guards in the try ... catch block, you can use this shortcut:

    try Expr
    catch
      ...
    end
    
  • To catch all exceptions, one needs to pattern-match _:_ in catch-block. Matching bare _ will assume catching only throws

General notes

  • Use export_all in development only. This is for two reasons. First, when you come to read your code later, you’ll know that the only important functions are the exported functions, and all the other functions cannot be called from outside the module so you can change them in any way you like, provided the interfaces to the exported functions remain the same. Second, the compiler can produce much better code if it knows exactly which functions are exported from the module.
  • Use match operators to avoid rebuilding atoms and other constructs that you’ve matched in the method signature:

    fun1([{user, Name, Age} = U|T]) ->
      ...
      f(U),
      ...
    
  • Data types ordering is as follows (any number is less than any atom etc.:

    number < atom < reference < fun < port < pid < tuple < list < binary
    

As I move on, I’ll be posting more of these self-notes…

Installing Command-T Plugin in MacVim

Recently I’ve switched to MacVim from Textmate, and done so for a couple of reasons:

  • I use Vim remotely when administering servers all the time, and practicing it locally makes me much more productive.

  • Vim is evidently more mature than Textmate. With the whole Open Source ecosystem around it, tons of plugins and superior text-editing features, it leaves Textmate far behind.

Of course, programming doesn’t happen in the editor, but you have to transfer your thoughts to the screen somehow and that’s where your tool chain comes into play. I found that after several days of practice MacVim started to feel at home and the fact that I keep discovering shortcuts makes it a game where you unveil more and more hidden goodness. Love the feeling…

RubyC 2011

RubyC is officially over and so I’d like to share my thoughts on the event – what went well and what desired to be better.

Let me start by saying I have a tiny bit of prejudice here. I lived in Australia, and the fact that three of key speakers were from the AU meant a lot to me. I met Pat before and so pairing with him and other mates in the park for two days would be as exciting as hearing them out in the conference room. Having this said, here’s what I liked and didn’t like that much.

Cron Jobs Monitoring With Monit

Monitoring services, CPU, disk and memory usage has become a common practice for IT companies big and small, but what often overseen are Cron jobs. (If you are unfamiliar with Cron, it’s the standard *nix way of running periodical tasks.) In this short post I show one simple way just to get you going.

The Ruby 1.9 Walkthrough

A couple of posts ago I wrote about the HTML5 Mobile Pro desktop edition the guys were doing a poor job with. Today I’d like to show what I consider a really good product.

I’m a big fan of everything that’s even remotely connected with Ruby. The language turned my world upside down and gave me my present life full of freedom, fun and amazing opportunities. That’s why when one of the guys I know pointed out that Peter Cooper is working on The Ruby 1.9 walkthrough, I (a bit skeptically) went to look around. Our ways didn’t cross too often, but he curated one of the rounds on Ruby Challenge and runs Ruby Inside, so he felt authoritative enough for the job.

It turned out to be right.