Learning Scala -- my experience and advice

Learning a new programming language has been on my list quite sometime and fortunately, I got a chance to try out Scala for one of the projects at work a month back. So, its been a month learning Scala now. Though its too early to form an opinion about it, but I must say, it has NOT been an easy language to learn so far.  And I think one of the reasons for that is -- the language is way too expressive with pretty much all the constructs in the basket and that makes it a very powerful tool but a difficult one to master. So one advice, don’t get disappointed if you feel lost and be patient while you are learning it because it is going to take a bit longer than you expected, so hang in there!!!

As I am on this journey, I am discovering some useful resources everyday, so I thought of documenting them in a blog post. I plan to maintain this blog post as a living document.


If you are looking for a quick introduction to the language, I would recommend “Scala for Impatient” by Cay S. Horstmann. It is well written book with right balance of content explaining key concepts and their application. But if you are in for a long and patient read, Programming in Scala by Martin Odersky is a good read which explains the key concepts on a much more fundamental and detailed level. I am using both the books  and cross reference depending upon the details I want to dive in to a particular topic. I am yet to explore other books, but these two makes a good start. The good thing is that both of them are available in kindle version.

Video Lectures: 

If you like attending video course, then there is a very good course “Functional Programming Principles in Scala”  by Martin Odersky on Coursera. The course is divided in to video lectures of 10-20 minutes length, which makes them easy to consume.

Blogs/Online Articles: 

There are some really useful blog post out there about scala, but a few deserve special mention here.


Eclipse and IntelliJ Idea, both of them, have a good support for scala. In my case, I end up using IntelliJ for my day to day coding, but eclipse has a very good support for Scala worksheet, which I find really handy to try and learn new concepts. For some reason, scala worksheets does not work in IntelliJ Idea. I also use Vim for some quick editing etc and I found vim-scala plugin pretty handy.

Let me know what has been your experience learning Scala and chime in with links to resources which you found useful.

Drake - Make for Data!!!

Very recently, I started my journey in big data world and its exciting time to see new technologies popping up these days. I came across this really handy tool which I used in my project at work. Its called Drake, Make for Data as factual folks call it. You can read more about it in their blog post . 

I will write up a detailed post about how I used Drake to simplify my life at work but in the mean time, I wanted to share this very good introductory video by Artem Boytsov the author himself.

I would highly recommend it to anyone to add it to his toolkit if you are dealing with complex data workflows in your project. Have a look and let me know what you think!!!

Highlight Call to Action by giving a small Wiggle

Today, when I was trying out a new application bufferapp, I noticed something very interesting in their interface which immediately caught my attention. Although it is a tiny little detail about the interface, but I think it is one of the very important ones especially in context of a web app, so I thought of writing it down.

If you have used bufferapp, you will notice following things (shown in the picture below):
  • High level navigation at the top (header section).
  • Notification with some call to action right beneath the header section (in this case Install Chrome extension)
  • Main content section which is about buffering your tweets. 
The detail that I wanted to point out was, when I am navigating from bottom section to the header section, I would see "install" button wiggles for a moment. Now that momentary wiggle made me pay attention and read that message to understand install what ?  It felt exactly like as if you are navigating through your office corridor and someone calls you out to show you something. So this "wiggle for .2 seconds" was almost like someone calling out to make sure I read that message. You can see this in action here in modern browsers specifically Chrome, Safari or Firefox.

You will see this layout pattern in lot of places where application is trying to communicate something important to the user and would like him/her to take an action like in this case Buffer wanted me to install their chrome extension as I was using Google Chrome. Typically, users develop a blind eye to some of these layout patterns and it becomes very challenging for the interface designer to craft an experience which could draw user's attention to an important action.

These simple but important details separates a good design from a great design. In rest part of the post, I would throw some light on implementation details for this wiggle effect. Most of the modern browsers support animation through CSS3 these days.

keyframe definition for wiggle effect would look like this: (omitting the browser prefix for better readability)
@keyframes wiggle {
   0% { transform: rotate(2deg); }
   50% { transform: rotate(-2deg); }
  100% { transform: rotate(2deg); }
Think of keyframes directive as defining a animation function which will be called later on elements like buttons in our UI. We implement wiggle function by first rotating the subject by 2 degree anticlockwise, then to 2 degree clockwise and then rotate it back by 2 degree anticlockwise.

Let say our html markup for flash message looks like this:

<div class="flash_message">
     Install Now
Inorder to invoke it on a mouseover of the flash message block. Our invokation code in CSS would look like this:

.flash_message:hover a.call_to_action {
  animation: wiggle 0.2s;

Here is a live demo which I have hosted on github and the code can be found in wiggle repo.

This is a classic example of applying CSS3 techniques at the experience layer in your application. Let me know what you think of such experiences, drop a note below in the comment section or tweet me @_sunil_ .

Asynchronous IF implementation using JQuery Deferred Objects

I ran in to a situation in a project recently where I got an opportunity to use JQuery Deferred Objects. I wanted to implement an Async IF operation. Async IF operation is like watching a condition to become true (may be in future) and then invoking something when it becomes true. So this is essentially an "if" operation in async mode and your logic might involve watching more than one condition to be met.

For example:
Consider a demo web application where colored boxes (red or green) can appear on the screen depending on some logic or user action. We want to do certain things if a red box appears or a green box appears or both of them appears.

Let say we we want to do something when a red box becomes visible on our screen, so it will look something like this:
async_if is_box_visible, "red"
whenever it is true, then do_something

Another one, let say, we want to pop up blue box when we see red and green boxes are visible on our screen for the first time, so it will look something like this

async_if is_box_visible, "red"
async_if is_box_visible, "green"
whenever both the above are true, then inject blue box

So here we are composing operations which involves some async operations because we do not know when these conditions will be met. This is exactly where Deferred objects shines.

I prepared a demo for the above scenario and rest of the part, I will walk you through some samples. I have shared the entire code in github repository at async_if. You can see the demo here.

Async IF Operation: It is a generic construct which takes two arguments:
  • A function which checks a condition (lets call it checker function henceforth)
  • Arguments to be passed to the checker function
  • timeout (milliseconds) for reporting failure if condition is not met within timeout milliseconds

Async IF operation returns an object of type Promise which can be used to compose higher level application logic using jquery's operations around deferred objects. Following javascript code is generated from the coffee script async_if.coffee.

Use of Async IF Operation: In code below, we are trying to build logic which is when red and green box both appears on the screen, inject a blue box on the screen. Now appearance of red and green box is purely async which will happen on user action like "injecting a box" by clicking on buttons in the demo application.

You can read more about when operations here (http://api.jquery.com/jQuery.when/)

Let me know your thoughts about this construct. Drop a note below or contact me at @_sunil_

PyCon India Presentation on Redis And Python

Pune (India) witnessed probably the second biggest PyCon event this year. It was third PyCon to be held in India and was very successfull event.

I took a comprehensive tutorial on Redis & Python subject. I am embedding the slides from the same session below.

Hope you will find it useful. Ping me @_sunil_ if you have any questions.

Another Redis use case: Centralized logging

Log analysis has become a difficult task in our production environment at work because logs are distributed on different machines and in different files. So, we wanted all the exception logs from all of our apps to be tracked centrally and viewed in single console. 

And once again, we found another good use case for Redis. Our strategy is to dump all of critical logs in to a Redis List and have a background worker which continuously pulls logs from the Redis List and write stuff in log file.

As we use python for all our backend work, I quickly wrote a Log Handler that can dump log messages in to Redis. 

So RedisLogHandler class looks like this:

To hook up this RedisLogHandler to your application logger, all you need to do is following:

I quickly hooked it up to our all of our background jobs in celery and I can see it working. Let me know what you think of this solution.

Redis Lua Scripting

 Redis is one of my favorite data storage platform and I won't miss a single chance to use it wherever I can. One of the biggest strengths of Redis has been that you can define your data modelling in the most natural form like key-value/hashes/lists/sets. I prefer thinking of data storage in form of list, hashes, sets instead of tables :).

Redis provides a very good set of APIs(commands) which pretty much allows majority of the operations you would like to perform on these data types in a single command. 

But in the past, I encountered situations where solution design required the following:
  • Executing more than one Redis commands.
  • Outcome of one command would determine whether to run other commands or not.
  • Atomic execution of 1 & 2 [Transactional].
  • Contention environement for such operations, so optimistic locking schemes wont work.
Situations like above pretty much made me turn away from Redis in the past. 

This is going to change now. About a month ago, @antirez (creator of Redis) released Lua-Scripting support in a different branch by hacking over a weekend (that shows how terrific hacker he is).

Lua-Scripting support Redis pretty much solves majority of the problems of the nature described above. This has great potential and can't wait to see what community do with it.

@antirez's post about scripting support provides good details. You can read the blog post here (http://antirez.com/post/scripting-branch-released.html).

I spend sometime today to try out scripting, so thought of doing a quick write up about what I did today.

I started with simple goal of implementing two new commands zpop and zrevpop for sorted set data type using scripting.

1. ZPOP: This will allow popping out element with lowest score from a sorted set.
2. ZREVPOP: This will allow popping out element with highest score from a sorted set.

Follow these steps to get scripting version of Redis running on your machine.
2. cd redis
3. git checkout scripting
3. make
Now you should all the binaries ready in the src folder (redis-server and redis-cli). Run the server by running redis-server binary.

You can take a quick crash course on Lua for writing some basic script. Follow this link (http://www.lua.org/manual/5.1/).

ZPOP Implementation:
Redis implements redis.call interface to invoke redis commands from Lua code. Here is Lua script for zpop command.

Redis server implements new command "eval" to invoke lua scripts. Quick syntax goes like this:
EVAL <body> <num_keys_in_args> [<arg1> <arg2> ... <arg_N>]
You can test your lua script using redis-cli program. 

Lets populate sorted set named zset by inserting a, b, c with scores values 1, 2 and 3 respectively.
./redis-cli zadd zset 1 a
./redis-cli zadd zset 2 b
./redis-cli zadd zset 3 c

Here is command to test zpop.lua file. You should see 
./redis-cli -p 10000 eval "$(cat path-to-zpop.lua-file)" 1 zset
1) "a"
./redis-cli -p 10000 eval "$(cat path-to-zpop.lua-file)" 1 zset
1) "b"
On the similar lines, zrevpop can be implemented using following lua script.

I am going to do a follow up post with some complex examples to demostrate the true potential of Lua scripting.

Thanks @antirez for keeping this project on the edge by innovating at all the levels of the project. 

CoffeeScript with Backbone.js Example

I have been spending sometime researching development strategies for building heavy Javascript application. I have been having some fun playing with backbone.js and coffeescript. I did not find some good tutorials that explains combine usage of these two awesome projects, so I thought of writing one myself.

Before we start, here is a quick intro about Backbone.js and Coffee Script from from their main website itself.

1. Backbone.js: Backbone supplies structure to JavaScript-heavy applications by providing modelswith key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.

2. Coffee Script: CoffeeScript is a little language that compiles into JavaScript. Underneath all of those embarrassing braces and semicolons, JavaScript has always had a gorgeous object model at its heart. CoffeeScript is an attempt to expose the good parts of JavaScript in a simple way.

In order to demonstrate usage of backbone.js and coffeescript, let me first describe you a simple example that we going to build today. You can see the final build example here. Basically, we will build a color-box configurater which takes size and color for square box to be drawn. Changes in the configuration should reflect in real time on screen. This type of functionality you see in any visual designer part of an IDE.

Coffeescript and Backbone.js example


Example consist of following parts:

1. Configuration Model: which basically contains three configuration properties, i.e. height, width and color of the box. We create ConfigModel class which inherits Backbone.Model class and its initialize method (which gets invoked when an instance is created) sets up the default values of the three properties. 


2. Configuration View: This view contains two input text boxes which are initialized with default values. This view is responsible for capturing inputs and updating the underlying model. We define two form fields color-input and width-input for taking color and size of the square. Note the identifiers (color-input/width-input) of these two elements and identifier (config-input) of the config container as they will be used coffee script part of the code.

ConfigInputView class defines the presentation layer of the configuration input and it inherits from Backbone.View class. Initialize method creates a linking between the underlying model with this instance of ConfigInputView class. Events field of the class basically binds keyup events on both the input boxes to updateConfig function. updateConfig function simply updates the underlying model's three properties height, width and color. Note the thick arrow here "=>" which specifies that updateConfig function should be invoked in context of this instance of ConfigInputView. 


3. ColorBoxView: This view is responsible for presenting a single colored box as per the configuration. This view responds to changes on configuration model. Another point to note here is there can be more than one instance of such views and all of them respond to changes in the configuration. We define ColorBoxView which inherits from Backbone.View. Initialize method of the view create a compiled class from a template which is also defined below. We define a method render which simply redraws the content of color box by generating HTML for the containing element from current configuration. Note we bind render method to be invoked on any change event on the underlying model. The underlying model is config model which gets changed on every keystroke in any of the two input boxes. Render function again defined using thick arrow "=>" makes sure that render function is invoked in context of ColorBoxView instance.

This is the HTML template file that we use for a color box presentation. We are using jquery template library. Two data pieces ${width} and ${color} are filled in the render function to generate the template.


4. Controller: Controller is kind a core which binds everything together basically the above three components together. Controller class inherits from Backbone.Controller class and instantiates one instance of ConfigModel, ConfigInputView each and five instances of ColorBoxView. Each of the ColorBoxView gets ConfigModel instance as underlying model.


Lastly we define a global init method which create an instance of ColorBoxController and is invoked on documentLoad event of JQuery.

I am sure you will have a few questions about the post, please comment below or tweet me @_sunil_

Facebook Python Library Documentation

This weekend , I spent sometime in exploring documentation tools available in python. I think Sphinx is the best documentation project I have ever come across. Python Docs itself uses Sphinx for documenting Python.

I thought of trying Sphinx for documenting official Facebook Python SDK. After spending some serious 4-5 hours, here it is:

Hope you guys like it. I plan to keep it updated with more writeups.


Shit happens, It does!

shit happens

Picture by Anant S.

We've all had embarrassing moments in our carrer where they involved inadvertently wreaking havoc on a production system. When it happens; for a second you (so desperately) want to believe it didn't. You will be so afraid to even cross-check that it actually happend. 

Github went through an outage yesterday and Chris was brave enough to reveal how it happend, then hacker news post generated a good buzz around the subject. While reading comments on both the threads, I hand picked a few interesting stories about production mishaps. Here they are:

seldo: My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who.

pixdamix: Mine was for a French social networking site 4 years ago. They used to send mails everyday to say "hey look at the people who you might know". The links on the mail would automatically log the user on the website. When I sent the code live it took 2 days (and more than 50000 mails to found out that when I sent a mail to person Z about person Y the link logged in Z ON Y's account.

SkyMarshal: sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.

lambda: The Dreamhost case in which they typed the wrong year in their billing code; charged many of their users for an extra year of service, to the tune of $7.5 million.  (http://blog.dreamhost.com/2008/01/15/um-whoops/). 

JabblesWhen the google engineer added "/" to the list of bad URLs, thereby marking every single website (!) as possibly dangerous. http://googleblog.blogspot.com/2009/01/this-site-may-harm-your-computer-on.html Now that would be a bad feeling.

dacortAt backdoor, the outage was caused by a test script that was meant to refresh a test database when it was inadvertently run against our production systems.  This script truncated 79 gigabytes of data in a matter of seconds. http://www.bigdoor.com/blog/bigdoor-api-service-has-been-restored/

joeybakerI ran datamapper auto_migrate! against the company blog mysql db with 5 years worth of posts and my sysadmin soon realized we didn't have a backup. Three fun filled days of Google cache scraping got us back up and running.

Would love to hear about your production mishaps if any :).