Backing up has never been easier with a little Ruby gem – backup. It offers huge array of features including various database adapters, flexible filesystem archiving, compression, encryption, backup destinations and synchronization. Every option is explained in its corresponding section in details.

It’s not a secret the most valuable asset of any application is its data and you have to protect it at all costs. In the following series of posts I would like to share several strategies I’ve come across over the years.

Backup Ruby gem. Not to waste much time on theory, in this post I would like to run through the real scenario of backing up a snapshot of directories and the database snapshot to Amazon S3. It doesn’t matter if you aren’t into Ruby. Ruby is currently on almost every Unix box by default, so you could start using the gem straight away.

Incremental backup with RSync will show you how to quickly sync your local data to a different location (Dropbox folder or external drive, for example). Nothing out of ordinary, but has its uses, as well as serving a nice introduction to more curious stuff.

Theory of backups rotation will talk about rotating your backups with minimum disk usage, while maintaining completeness. Sorry Windows users, it’s Unix-specific.

Amazon EBS snapshots concludes the series of posts with a very practical scenario that you could apply today if you are an Amazon EC2 user.

Hope you’ll enjoy it!

I figured I might want to make a quick post here for those who follow this blog, and not my Twitter account. Recently (on Jan 24, 2011) Kate and I became parents. We named our first-born “Max” and he is a very active little buddy.

Ok… have to go. He’s giving some hard time to his mom. :)

By default, Mongoid generates long hash-like keys from BSON that incorporate time, machine ID and counter. Pretty unique, but user unfriendly (for example, 47cc67093475061e3d95369d).

To solve this, Mongoid lets you define a custom composite key on the document and during the creation of the key it escapes special characters that may be found in the original string. So for example, ‘~’ is converted to ‘-tilda-’ and ‘-’ is converted to ‘-dash-‘. Yes, I don’t like it as well as I don’t like ‘_’ to be converted to ‘-undscr-‘. It looks pretty ugly in the URL.

So here’s a little trick if you want to make it replace all non-alpha and non-digit characters with dashes:

You need to put it in the initializer. I do this in config/initializers/mongoid.rb.

It has become an established practice these days to have a video tour on the application web site. When the concept of the app is new it is supposed to help users to figure it all out a great deal.

In the case of 9Cells, we have very few conversions so far. People don’t even try to click on the “Create Survey” link, and my best guess is that’s because they don’t see the meaning and value in the application.

This is primarily the reason I decided to invest some time and effort in creating a brief end-to-end tour focusing on clarity and the ease of use.

The script

I started with the script that would reflect a workflow of a typical user:

• Click on the “Create Survey” link
• Log into Twitter
• Create the survey
• Load credits into account
• Start the survey
• Monitor the progress

Next I ran through the workflow and imagined what I would be interested in knowing at each step and added necessary comments and explanations to the script.

Video

To record the video I tried several tools. On Mac we have this free QuickTime Player that is capable of recording the video from screen in real-time. It’s absolutely fantastic, but lacks a couple of features I needed for this recording:

• Limiting the area to a single window dimensions
• No click indication

Yes, I can crop the area afterwards to limit the view, but my quick tests showed the results are usually crappy, unless I go nuts with hardcore semi-console tools.

After snuffing around, I found a very nice and simple tool — Screenflick — an all-in-one recording solution that sits in the status bar and does just that — records, records, records. It’s packed with all essential features (click indication, keypress popups etc) and very reasonably priced.

Lessons learned during the recording:

• Don’t move mouse cursor around unless it’s necessary. If you need to insert a few scenes later or need to stretch one for a bit longer to fit in all explanations, you will end up with jerky mouse cursor movements.
• Don’t rush. Take your time on each screen. You will be able to remove unnecessary frames later. It’s way harder to add them.
• Don’t scroll the screen unless necessary. Again, it makes harder to cut frames and insert new ones. If you need it, do it. If the hands are just itching, sit on them.

For the video mastering I chose iMovie app bundled with every Mac. It does the job relatively good and it’s free.

Audio

I’m not a native speaker and currently live outside English speaking countries, so that’s not an option for me to either narrate myself or find someone to do that for me. At least, it’s not THAT easy. That’s why I decided to use the “Alex” voice that comes with every Mac running Leopard. Saying whatever I need is as easy as running “say” command in Terminal. And what’s great about it is that I can run it hundreds of times to get it right — it won’t grow tired or pissed.

After all phrases were recorded to files, I started aligning them with video in iMovie. As expected, sometimes there were too much video, and sometimes I had to stretch it to let the narrator finish the phrase. All in all, it went surprisingly well, and I even enjoyed the process.

Embedding

With mastered video in hands I needed to put it on the front page of 9Cells. As usual, there are millions of options as to where and how to display the video. At first, I chose to put it instead of the gallery, and so had to reduce the size of the picture from 640x480 to even smaller 480x360. The result was barely viewable, and thus I made a step back and decided to use one of the lightbox popup plugins. I chose Fancybox. Why? It’s easy to configure, does what’s necessary and has callbacks. I’m sure there is at least a dozen of them falling under this description, but this one didn’t give me a single chance to regret it.

To play the video there’s one almost defacto tool — Flowplayer. It’s on Flash (yeah, I know about iDevices), but it works for the huge range of video formats, has hundreds of plugins, easy to script and incredibly flexible.

Finally, I took 10 minutes to create a “Video tour” badge in Photoshop.

Conclusion

The results of this effort are currently on 9Cells. Let me know what you think. Any feedback is warmly welcome.

Just released a fully functional early BETA of 9cells.com – the service that lets you get votes for up to nine images in one compact survey.

• Designers: Check which of your logos, designs, templates people like the most
• Photographers: Find your best composition, colors etc
• Eveyone else: Not sure which pic to print and hang on the wall? Which of the flowers to send to your mom?

There’s plenty of uses!

Check it out and let me know…

Oh, and if’t not too much to ask, please tweet about 9cells.com and use #9cells hashtag.

There are several ways to execute a shell command from your Ruby script. They give you the output and exit code which is often enough. But what if you need more than that? Recently I needed to run the command that prompted for a password. Here’s my simple solution to this – you give the command as usual and then a Hash of matchers-responses. Every line in STDOUT will be analyzed and if it matches any of these guys, the response will be sent to the STDIN. As simple as that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

require 'open3'

def system_with_prompt(cmd, responses = {})
  buffer = []

  Open3.popen3(cmd) do |stdin, stdout, stderr|
    while line = stdout.gets

      # Find the response
      responses.each do |k, v|
        match = case
        when k.is_a?(Regexp)
          k =~ line
        else
          k.to_s == line
        end

        # Respond if found
        if match
          v = v.call if v.is_a?(Proc)
          stdin.puts v.to_s
        end
      end

      buffer < < line
    end
  end

  buffer.join
end

# - you get your output as usual
# - you can feed many prompt-response pairs in hash
# - prompt matcher can either be a string or a regex
# - response can be either a string or a lambda / Proc returning the result
output = system_with_prompt("command_with_password arg1", /password:/i => lambda { Time.now.to_s }, 'Another Filed' => '...')
puts output

RSpec has many great features that makes the spec’ing life easier, and one of them is the objects letting. Here’s what it is briefly:

1
2
3
4
5
6
7

describe Album do
  let(:album_with_image) { Factory(:album) }

  it "should have cover image" do
    album_with_image.cover_image.should_not be_nil
  end
end

As you can see, you are just defining the name of the object and how it can be initialized. Later in the tests you use it as a local variable and it’s always the same object inside each test. One positive side is that you can define as many objects as you need for your tests, but they will actually be initialized when and if you need them.

Now imagine this picture:

1
2
3
4
5
6
7
8

describe ActivityLogger do
  let(:post) { Factory(:post) }

  it "should log creation of a post" do
    post # creating new post
    Log.should ....
  end
end

Here we need to access “post” for RSpec to create an instance for us. That’s not very readable, right? I basically want it to look:

1
2
3
4

it "should log creation of a post" do
  create_post
  ...
end

Here’s how I do it without hard-coding the methods for all possible factories that I may want to define (from “/spec/support/helpers.rb”):

1
2
3
4
5
6
7

def method_missing(name, *args)
  if /^created?_(.+)$/ =~ name.to_s
    send $1
  else
    super
  end
end

Now I can address “post” as “create_post” and “created_post”. Of course, it creates a “create*” and “created*” aliases for every method in the scope, but I don’t worry about that since the naming is very specific.

Love it, hate it?

Recently, the project I was working on was deployed for public testing and one particularly interesting vulnerability was discovered that let the attackers execute arbitrary code on the server with the application owner permissions. I would like to share this information for you to avoid the same trap.

In this project we use Paperclip and needed a custom processor to perform the encryption of the uploaded files. Paperclip has a very convenient interface for processors that you can implement and then use to alter the uploaded files – perform OCR, rotate images, archive data, compress JS / CSS and do a million of other useful things. When the file is uploaded, Paperclip saves it in a temporary location (usually /tmp) and lets all configured processors to work their magic on that file. Later, when and if the processing succeeded, the file is moved to a permanent location and is given the correct name.

The danger lies in the naming of the temporary file. Here’s how the temporary file is named:

1
2
3
4
5
6
7

class Tempfile < ::Tempfile
  # Replaces Tempfile's +make_tmpname+ with one that honors file extensions.
  def make_tmpname(basename, n)
    extension = File.extname(basename)
    sprintf("%s,%d,%d%s", File.basename(basename, extension), $$, n.to_i, extension)
  end
end

Paperclip, when saving the uploaded file, does it like this:

1

tempfile = Paperclip::Tempfile.new("stream" + File.extname(name))

(where the “name” is the name of the originally uploaded file – the one that is the name of the file you choose to upload)

The resulting temporary file that is created has the format “stream,PID,N.EXTENSION”, where “N” is counting from 0 for each new file and “EXTENSION” is the original extension of your file. We are getting close, bear with me.

Now lets assume that you want to process the uploaded file in a way and you have the command-line tool for that named (surprisingly) “processor”, and so you write the following processor code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

module Paperclip
  class Encrypt < Processor
    def initialize(file, options = {}, attachment = nil)
      super

      @file           = file
      @recipient      = options[:geometry]
      @attachment     = attachment
      @current_format = File.extname(@file.path)
      @basename       = File.basename(@file.path, @current_format)
    end

    def make
      src = @file

      begin
        Paperclip.run("processor", "\"#{File.expand_path(src.path)}\"")
      rescue PaperclipCommandLineError
        raise PaperclipError, "couldn't be processed. Please try again later."
      end
    end
  end
end

Here’s where it all happened. In the line where we run the command-line tool, we give the name of the file as an argument, and that’s all fine for the name like “image.png”, but gets completely out of hand when someone sends the file with the name like “image.$( sleep 10 )”. In this later case, the temp file created by Paperclip and passed to your processor will look like this:

1

/tmp/stream,12345,0.$( sleep 10 )

… and when used in the command line, like this:

1

process "/tmp/stream,12345,0.$( sleep 10 )"

It will put the server in deep coma for 10 seconds. Now imagine if it’s not “sleep 10”, but “rm -rf ~” or anything else copying important stuff to publicly accessible places?

So what’s the cure? My solution was to check extension of the temp file and disallow anything that’s not what we want to accept. It’s simple and simple is what we need here.

Please let me know if you have a better way of solving this or found something I missed.

Today while playing with Amazon EC2 I installed Apache httpd 2.2.15 and discovered that previously working SSL module configuration no longer works. The server doesn’t start and the error log has this:

1

[Sun Oct 03 16:37:04 2010] [error] Server should be SSL-aware but has no certificate configured [Hint: SSLCertificateFile] ((null):0)

I know for sure that the configuration I have worked, but for some reason it stopped. Instead of downgrading I decided to find the truth.

It appears the newer version of httpd has stricter SSL configuration rules. Now you must have SSLCertificateFile and SSLCertificateKeyFile options in your VirtualHost definitions, so they should look like this:

1
2
3
4
5
6

<virtualhost *:443>
  ...
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
</virtualhost>

Hope it helps as I see tens and hundreds of posts from people everywhere puzzling over this.