Setting up a Data Warehouse with AWS Redshift and Ruby


Most startups eventually need a robust solution for storing large amounts of data for analytics. Perhaps you’re running a video app trying to understand user drop-off or you’re studying user behavior on your website like we do at Credible.

You might start with a few tables in your primary database. Soon you may create a separate web app with a nightly cron job to sync data. Before you know it, you have more data than you can handle, jobs are taking way too long, and you’re being asked to integrate data from more sources. This is where a data warehouse comes in handy. It allows your team to store and query terabytes or even petabytes of data from many sources without writing a bunch of custom code.

In the past, only big companies like Amazon had data warehouses because they were expensive, hard to setup, and time-consuming to maintain. With AWS Redshift and Ruby, we’ll show you how to setup your own simple, inexpensive, and scalable data warehouse. We’ll provide sample code that will show you to how to extract, transform, and load (ETL) data into Redshift as well as how to access the data from a Rails app.

I created a tutorial for Redshift and Ruby here and wrote about it on our new engineering blog.

Controlling Unity with Leap Motion

Screen Shot 2014-09-04 at 11.59.16 AM


I’m still waiting for the Oculus DK2 to ship so I’m doing an open source project to find a more intuitive way to translate, scale, and rotate objects in the Unity. If you’ve seen the Iron Man movies or Elon Musk designing rockets with his hands, you’ll understand what I’m going for. Manipulating 3D objects using keyboard and mouse is quite limiting. If we can improve on this, it would allow us to create content for VR faster.

We’ll be using the Leap Motion, which ships in 2 days from Amazon :-). The plan is to create a custom Unity editor window that will communicate with the Leap Motion sensor. We’ll then connect the hand movements directly to the translation, scale, and rotation controls for the currently selected GameObject. Feedback is always welcome. Here’s a link to the source on Github:


Rails Google OAuth2 Tutorial

Google recently deprecated OpenID 2.0 authentication, which I used to authenticate users via Google Apps for internal projects like our Dashboard. In a couple of months, it will just stop working so I’ve been converting projects to use OAuth 2.0. Google login is pretty convenient, especially if your team is on Google Apps. The conversion process was very annoying so I hope this tutorial saves you time.

First, we’ll need to setup a new project in the Google Developers Console.


Next, enable the “Google+ API”:

google plus

Go to “APIS & AUTH > Credentials” and click “Create New Client ID”. You’ll need to configure the origins and redirect URIs for every domain you need. I’ve configured it for development and for Heroku so you can see a live demo.

client settings

You should now have a CLIENT ID and CLIENT SECRET. Let’s put them in your shell startup script so that your app can access them. We do it this way so that you don’t check in sensitive information into your source code.



Now we can run the example:

source ~/.bash_profile
cd ~/Sites
git clone
cd google_oauth2_tutorial/
bundle install
bundle exec rake db:setup
bundle exec rails s

This loads your shell startup script, grabs the source code, setups up the database and starts the app. If all went well, you should be presented with the Google Login screen. After logging in and approving the app permissions, you should see “You are logged in via OAuth 2.0 as <your email>!”.

More Details

This tutorial uses the Omniauth gem, which makes it easier to provide multiple ways for users to authenticate into your app. You specify what you want your app to allow as individual “strategies”:


Rails.application.config.middleware.use OmniAuth::Builder do
provider :google_oauth2, ENV['GOOGLE_CLIENT_ID_TUTORIAL'], ENV['GOOGLE_CLIENT_SECRET_TUTORIAL'], {scope: 'email,profile'}

Tip: If you want to use this for your Google Apps domain, simple pass an additional parameter:

provider :google_oauth2, ENV['GOOGLE_CLIENT_ID_TUTORIAL'], ENV['GOOGLE_CLIENT_SECRET_TUTORIAL'], {hd: '', scope: 'email,profile'}

The whole flow can be confusing so make sure you reference the Omniauth documentation before trying to troubleshoot. I found that if you don’t fully understand the flow, it will be very hard to debug your code. However, once you do, adding other strategies like Facebook or Twitter should be much easier.


If you’re seeing “invalid client_id”, your environment variables are probably not set correctly. You can use the “printenv” command to verify if the particular terminal tab you’re running the server in has the right variables. If not, source your shell startup script again. If you’re seeing API permission errors, you probably forgot to enable the Google+ API. Google’s documentation has more detailed information on specific errors that may help. If all else fails, clear your browser cookies for localhost.


Why the Internet is Broken

Let’s face it: the Internets are a mess. In switching from Google Reader to Flipboard (since Larry’s pulling the plug soon), I’ve been forced to take stock of my online identity and content consumption habits. We can do better than this, tech people!

I am a thousand different special snowflakes

I’ve recently changed jobs. Naturally, I updated my LinkedIn profile in 30 minutes. But then my Twitter profile also needed to have my title and company changed. And then I noticed my blog is wrong as well. And finally I’m left feeling like the Internets are broken because I don’t know what else needs to be changed. These are the variations of my identity floating out there. This is easy compared to usernames and passwords. Some companies are trying to solve this problem in different ways like Gravatar with profile images and OnePassword for logins. Hoever, there’s a common stupid notion that a big player like Google or Facebook should own all of it. They fight to segregate and own versions of our online identities. I’m tired of these winner-take-all wars that hinder progress. The Gravatar model is the way to go. Profile information in one service that’s simple to use. Someone’s probably already built this for profile info but people have to jump onboard this free one-stop shop concept bandwagon for this to be the norm.

Digital Gluttony

Information is now cheaper than dogfood with horsemeat. There’s a lot of it and no one’s peddling the equivalent of Atkins or South Beach Diets…yet. There are primary sources like Techrunch and then secondary sources like Facebook, Twitter, LinkedIn, etc. There are curated feeds like those provided by the likes of Flipboard or even feeds of feeds like Alltop. I can get the same article from many different channels and pictures or videos to entice me to keep consuming. Very quickly I get a headache. And I haven’t even turned Push notifications on. Facebook used to impress me with their filtering but now it’s just an exercise in branded **** avoidance. Twitter’s branded **** isn’t as bad yet but they just point the firehose at you with no regard for human life. Startups like Flipboard look like they’re trying to do some filtering but so far it seems like they’re more obsessed about how to make everything look like a magazine (suggestion: not everything should be in magazine format).


Both are tough problems whose best solutions require innovations that companies and markets don’t excel at: standardizing services, sharing data freely, and working together towards a larger goal. Take Amazon as a microcosm of the tech world. Jeff Bezos sends out a memo. He decrees that all departments must now define APIs, share data with ease through these APIs, and use this infrastructure to create AWS. This is a pretty loose interpretation but bear with me. Now imagine if Facebook, Twitter, Google, etc. were merely departments of one large company that wants to help people manage their online identities and curate their 2000 daily calories of content. LinkedIn handles the initial update. My friend sees that I changed my job via Facebook a few seconds later only because he’s a close friend. Moo asks me if I want new business cards. Google switches my work email over to the new domain. Granted some of this would be creepy by today’s standards but it illustrates reasons why it’s all a mess. Wouldn’t it be nice for companies and markets to share a purpose or two?

What Startups Can Learn From Planetside 2


If you haven’t heard of Planetside 2, it’s about time you got acquainted. It’s a new kind of game described as MMOFPS (massively multiplayer online first person shooter). It’s currently in beta with a launch date next month but most everyone would agree that there’s no way in hell it will be ready by then. Me, I’m confident they’ll be fine because they’ve got elements startups need to emulate. 

Iterate, Iterate, Iterate

During the beta, their servers are up from 6am until midnight during the week and 2am until midnight on the weekends. What happens during those few magical hours? They’re looking at data (e.g. how many sniper shots were fired in grid location L5 against main battle tanks) They’re reading the thousands of posts on their forum to distill wisdom. They’re coding away like mad. Welcome to 1-day release cycles. Mind you, they’re not just making minor tweaks. One day, after players had complained about a primary game resource, they just removed it. They iterate on their product really quickly and aren’t afraid to make big changes.

Build a Relationship With Your Customers

If you heard “Sony Online Entertainment (SOE)”, you would probably think about stodgy old men sitting around, smoking cigars. I’m impressed by their tenacity when dealing with clients. Sure, they have a website and a forum and the social media accounts. But then they go beserk with weekly live streaming events called Friday Night Ops. The President of SOE, John “Smed” Smedley, routinely posts on the forums and at first I thought he was a dev. They have a youtube show where they talk about Planetside news and show off fan-produced content. They have personalities like Matt Higby and Margaret Krohn who really know the game and community and are everywhere. 

Go BIG or Go Home

Battlefield, Call of Duty, Halo–they all have limits for the number of players on a single player. Typically, it’s 32 or 64. On planetside, it’s 2000. That’s two orders of magnitude, folks. Website where you can see every player’s configurations and analytics? Check, and it’s free. Mobile and tablet apps? Check. Every game mechanic you’ll find in the other top FPS’s? Check. Graphics that will make players broke and computer makers ecstatic? Check. Usually, startups should focus on doing one thing really really well. But sometimes, you just need to go BIG.

Conflict is Natural and Good in a Startup

A recent HBR video spurred me to think about conflict. Society teaches us to avoid conflict, which is a decent general survival strategy; however, it benefits us to understand why it’s natural and good–especially in a startup.

It’s Natural

Conflict is borne from differences. As a people, we’re differentiated by gender, race, ethnicity, education, religion, politics, etc. It’s why we have such a rich variety of cultures and individuals. It’s how we get Picassos and Einsteins. But it’s also how we get serial killers. Each of us has a unique point of view and a rich history of experience that shapes how we think. In a startup, you’ll have a group of people who try to use the sum of this knowledge and experience to make decisions that will ultimately decide whether or not the company survives. The personalities of this group is self-selected from the passionate and risk-taking part of society. Furthermore, startups are usually starved for resources, which inevitably leads to hard prioritization decisions. If you don’t fight for your part, who will?

It’s Good

Without conflict, a startup team is missing voices. If people don’t fight for their thoughts and beliefs, the best solution to a problem may never have been presented at all. People are typically poor at making complex decisions individually. If you have an frontend engineer, a backend engineer, and a platform engineer all discuss a problem, they’ll attack it from different angles and bring a broader set of solutions. If everyone’s a backend engineer–well, it’s really about covering all the bases. 

Moving Forward

Granted, this is the optimal case. If you have conflict that focuses on personal grudges that taint every discussion, that’s not gonna help. If you have people who enjoy conflict for the sake of conflict, then watch out. If you a wide range of talents (A vs B vs C players), this can make it more challenging. They key is to realize *why* there’s conflict. If you know that the guy across the table doesn’t agree with you because your goals aren’t aligned–that could be an easy road block to remove. If you recognize that everyone’s fighting for the same finite resources–that could be opportunity to reach common ground. Whatever it is…understanding the other parties in the discussion is the key to making good come out of conflict.


What companies are Facebookers leaving for?

Now that Facebook has “quietly” IPO’d, it’s time for Suck to train his army of engineers on more meaningful pursuits. @elonmusk is a good role model. After Paypal, he’s been working on Tesla and SpaceX. Tesla has made electric cars sexy again and SpaceX just launched a Dragon. Granted, the movie “Social Network” was pretty good and it used to be fun thinking of Facebook as David fighting the Goliath, Google. But now, the rebels have become the Evil Empire. What companies are Facebookers leaving for?

Every Startup Needs a Gong

This is our third gong. Our first one was a few inches in diameter and sat on a desk. Our second one is about six inches in diameter–a mini version of this beast, which vibrates your soul if you stand too close. Gongs are a meme at Badgeville–a unique story about our company just like “Team Punishment” and Iceland. We hit the gong whenever sales closes a deal and the size of the deal determines the number of gongs (and now also what size gong you hit). Other companies also does this for sales. I hear New Relic gongs and sends a mass email to the entire company when a client goes premium (awkward and funny story around that). I love the gongs because they’re a tangible reminder of progress. As an engineer living in a world of 0-downtime releases and in the past weekly or even daily releases, there’s no party to commemorate the mailing of CDs with your software. Somewhere out there, some VMs turn over in a dark, temperature-controlled room and your SaaS product has been updated.

Gongs are one of the many ways companies develop culture and character. Our name is “Team Punishment”. The origin of the name is oh-so-random but it’s stuck nonetheless. Every quarter we design t-shirts to commemorate the evolution of “Team Punishment”–recent ones featured images from the Godfather and Scarface. I remember one quarter we had one w/ a japanese phrase translated as “team with no worthy enemies”. As we’ve grown to over 50 employees and moved 4 offices and now occupy a fifth across the street, it’s the little things like gongs and team names and backstories that pass on the culture, the essence of the company. The space-time continuum in a startup is compressed–years feel like months and weeks feel like days. It’s very easy to get caught up in Getting Stuff Done (GSD), but it’s these tiny details that help tell the story of your company.

Why Google is Dying to be More Social

It’s tiring to hear that Google doesn’t have “social” in its DNA. I left a comment on the article, but I must elaborate on this battle of epic proportions happening in the tech world.

Arrogance is a Ruse

Google is or can be perceived as being arrogant. It’s unlikely that a company that has transformed the world as Google has can avoid a tinge of arrogance. Hence, it is naive to think that Google’s attempts at “social”, namely Google+, are merely whimsical dalliances of an aging giant. I don’t think anyone at Shoreline realistically thinks Google+ will ever overtake Facebook. The arrogance is a ruse to throw us off the scent–the smell of deepening fear.

Beyond PageRank and Wide Open Spaces

Google’s stated mission is to “organize the world’s information and make it universally accessible and useful”. When what all the world knew was Yahoo’s labyrinthian category hierarchy, Google’s algorithm for indexing and searching the growing sea of information was a quite a paradigm shift. This approach is still dominant today but the sea itself is changing. The Internet used to give Google free reign to crawl and index. Today, with digital fortresses like social networks and pay walls, it’s become more and more difficult for Google to complete its mission. For a while, Google had a deal to incorporate tweets into their search results. As of late, the relationship is still strained. Then there was the whole Google-Facebook address book debacle. More and more, the world’s information has evolved from merely flat web pages to intricate graphs containing not only people but brands, topics, and even pets. Companies like Badgeville and lately Facebook as well, go further, building “behavior graphs” where the connections between the content are rich verbs like “watched”, “purchased”, and “performed”.

Social or Bust

Google’s frustration is apparent and with good reason. Think back to Google’s ordeal with the Chinese government last year. The situation is different, but the problem is the same: the world is not as open as Google likes. The government writes the rules in China and Google must operate by them. It took Google a while to realize that. The same battle rages on between Android and iPhone. For better or worse, Apple keeps tight control over its iWorld. Some people like that approach, but for the rest of the market, Android is bringing order to the chaos. Google+ is an attempt to break down yet another set of barriers in another arena. It’s the counterweight to Facebook and its unique way of looking at the world. Whether or not you believe what Google believes, you should respect them for sticking to the mission, even if they don’t always clearly articulate them. However, you really have to wonder how Google will organize the world’s information when it has no access to large portions of it. Or perhaps what will happen if Google stopped fighting these fights.