I’m a big fan of Sidekiq, but recently, one of my clients asked for hosting their new application on Amazon’s Elastic Beanstalk which imposed certain limitations, that I had to overcome to get my normal setup working. In this post I want to describe some of them.

I usually either use Sidekiq on the same node, or place it on a dedicated one to put demanding workers away from the main web app. It requires Redis to maintain the queue of tasks and the copy of application to be able to execute them.

Amazon’s Elastic Beanstalk is the framework with load balancing and autoscaling that lets you focus on your application, not infrastructure. The copy of your Rails application (in this case) is unziped and configured on each instance with Nginx and Passenger. If you need database, they provide RDS that runs MySQL (in this case).

It appears that the setup with a dedicated node for Sidekiq and one with Redis isn’t quite possible with current state of Beanstalk. There’s no distinction between nodes – every one of them is just like the sibling. At the same time, there’s a notion of a “leader”, being the first node handled during the deployment, but we can’t use it to identify a certain node reliably afterwards. It can be used for migrations or other one-off tasks during the deployment only, and can change from deployment to deployment. On top of this, the auto-scaler is free to terminate any instance during downscaling, and I read it that it leans towards older instances more often than not.

Here’s what I finally came up with:

• use RedisToGo as an external Redis database. They host it on EC2 in the same zone and so the latency is super-small.
• deploy workers on each web node. Even though it’s an unusual setup, it makes sence. As the load grows, the number of tasks to run is likely to increase as well, and so scaling workers makes sense.

To (re)start workers on the nodes, I placed this .ebextensions/sidekiq.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq":
    mode: "000777"
    content: |
      cd /var/app/current

      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -TERM `cat /var/app/support/pids/sidekiq.pid`
        rm -rf /var/app/support/pids/sidekiq.pid
      fi

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      bundle exec sidekiq \
        -e production \
        -P /var/app/support/pids/sidekiq.pid \
        -C /var/app/current/config/sidekiq.yml \
        -L /var/app/support/logs/sidekiq.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq":
    mode: "000777"
    content: |
      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -USR1 `cat /var/app/support/pids/sidekiq.pid`
      fi

Basically, after unziping (pre-init task 1) and setting up environment (pre-init task 2), we tell Sidekiq not to accept any more jobs. Then new app version is deployed, and finally (post-init task 50), we restart workers by first, stopping them, giving time to terminate, and starting over again.

Figuring this out took about two days of dirty time, and so hopefully, this will be of any help to others.

In my projects, traditionally, I use cron to schedule periodical maintenance tasks and then do all the heavy lifting in the background. Cron would start a Rake task that would, in turn, schedule a job. Then Sidekiq (or Resque, previously) pick it and take care.

Recently I became aware of increased CPU and memory usage on the background workers server. Quick research showed that Rake tasks started taking a great deal of resources. While we had just several that started once an hour or even less frequently, everything went smoothly. As we’ve got some minutely tasks, things slowly went out of hand.

It’s obvious that using a Rake task for scheduling a job is an overkill. All the task does is placing a record in the Redis database with job details, so why spending 30 seconds loading whole Rails stack? Unfortunately, I couldn’t find any mention of starting tasks from the shell, so I went into the redis-cli tool and checked the queue for what exactly is being pushed in there. That appeared to be a simply crafted bit of JSON.

Here’s what I run to push a job (Pulse class) with no arguments to a “default” queue:

redis-cli rpush queue:default "{\"retry\":true,\"queue\":\"default\",\"class\":\"Pulse\",\"args\":[],\"jid\":\"db:pulse-`date +%s`\"}"

The only part deserving a bit of details is the “jid” parameter. I use a simple notion saying which node the task comes from (db), what is it (pulse) and a simple unix timestamp to both ensure it’s unique and know when it happened. As simple as that.

You can imagine how fast is it compared to the Rake business.

Oh, it’s been more than half a year since the last post! I figured, if there’s someone still following my blog, they deserve this post.

So what’s up with us:

• Max quickly becomes a fully formed man with the character, good dictionary and is already capable of building 3 or 4-word sentences.

• a couple of months ago we finally found a place of our own – a gorgeous 3 bedroom apartment with a spacious living room, fully equipped kitchen and great location in relation to our families

• I’m working on three exciting projects for clients involving video and photo processing, native clients for Android and iOS, paid subscriptions with PayPal, and a ton of other curious technological advances

• … as well as taking two online Stanford classes on building a startup and finances

• Kate’s hoping to resurrect her hand made bags business one day

• We finally relaxed a bit and stopped saving money for anything big. It cut us some slack and gave well-deserved peace of mind

• Another winter season is coming and I’m anticipating lots of snowboarding, travelling and fun guiding Max through his second winter

• A couple of days ago I ran across the announcement of Ableton Live 9 and Push that are due this coming winter. Immediately started dreaming of resuming my musical workings. I’m mulling over many enjoyable ideas to put on musical canvas eventually, so more stuff to come soon

That’s about all I have for you, fellas. I’m hoping to write more often, but hey, I hope it each time.

Until next time!

I’m using RSpec in most of my recent projects. In fact, whatever new I start gets RSpec installed by default. Recently I began using Sorcery gem for authentication. It’s a small and non-intrusive solution that I find very well-fitted for how I do things. If you are using Sorcery, you may be glad to find out that it’s possible to make your whole functional testing suite almost twice as fast.

Yesterday, while I was reading a book on Erlang, I’ve got this idea to devise and implement prime numbers finding algorithm without looking up anything. So I did. But then figured my friends may also want to get their feet wet and so I tweeted about the “challenge” and got 3 submissions (2 in Ruby and 1 in Erlang). Later, I discovered that a couple more would have wanted to try but never saw my tweet. Well, it’s a shame.

I’m reading excellent “Programming Erlang” by Joe Armstrong and find some new stuff I haven’t seen in “Erlang programming” by Francesco Cesarini. This is largely a self-note post, so take it with a grain of salt.

Lists

• If your code prepends elements effectively reversing the order of the original list, you need to call lists:reverse(...) as opposed to concatenating lists (L ++ [ X ]). The reverse call is way more efficient and even though it’s defined in lists module, when VM sees the call, it invokes an internal version.

Exception handling

• The value of the after block is lost
• The after block can be omitted

• If you don’t need any guards in the try ... catch block, you can use this shortcut:

  try Expr
  catch
    ...
  end
  

• To catch all exceptions, one needs to pattern-match _:_ in catch-block. Matching bare _ will assume catching only throws

General notes

• Use export_all in development only. This is for two reasons. First, when you come to read your code later, you’ll know that the only important functions are the exported functions, and all the other functions cannot be called from outside the module so you can change them in any way you like, provided the interfaces to the exported functions remain the same. Second, the compiler can produce much better code if it knows exactly which functions are exported from the module.

• Use match operators to avoid rebuilding atoms and other constructs that you’ve matched in the method signature:

  fun1([{user, Name, Age} = U|T]) ->
    ...
    f(U),
    ...
  

• Data types ordering is as follows (any number is less than any atom etc.:

  number < atom < reference < fun < port < pid < tuple < list < binary
  

As I move on, I’ll be posting more of these self-notes…

Recently I’ve switched to MacVim from Textmate, and done so for a couple of reasons:

• I use Vim remotely when administering servers all the time, and practicing it locally makes me much more productive.

• Vim is evidently more mature than Textmate. With the whole Open Source ecosystem around it, tons of plugins and superior text-editing features, it leaves Textmate far behind.

Of course, programming doesn’t happen in the editor, but you have to transfer your thoughts to the screen somehow and that’s where your tool chain comes into play. I found that after several days of practice MacVim started to feel at home and the fact that I keep discovering shortcuts makes it a game where you unveil more and more hidden goodness. Love the feeling…

RubyC is officially over and so I’d like to share my thoughts on the event – what went well and what desired to be better.

Let me start by saying I have a tiny bit of prejudice here. I lived in Australia, and the fact that three of key speakers were from the AU meant a lot to me. I met Pat before and so pairing with him and other mates in the park for two days would be as exciting as hearing them out in the conference room. Having this said, here’s what I liked and didn’t like that much.

Monitoring services, CPU, disk and memory usage has become a common practice for IT companies big and small, but what often overseen are Cron jobs. (If you are unfamiliar with Cron, it’s the standard *nix way of running periodical tasks.) In this short post I show one simple way just to get you going.

A couple of posts ago I wrote about the HTML5 Mobile Pro desktop edition the guys were doing a poor job with. Today I’d like to show what I consider a really good product.

I’m a big fan of everything that’s even remotely connected with Ruby. The language turned my world upside down and gave me my present life full of freedom, fun and amazing opportunities. That’s why when one of the guys I know pointed out that Peter Cooper is working on The Ruby 1.9 walkthrough, I (a bit skeptically) went to look around. Our ways didn’t cross too often, but he curated one of the rounds on Ruby Challenge and runs Ruby Inside, so he felt authoritative enough for the job.

It turned out to be right.