Setting up a Data Warehouse with AWS Redshift and Ruby


Most startups eventually need a robust solution for storing large amounts of data for analytics. Perhaps you’re running a video app trying to understand user drop-off or you’re studying user behavior on your website like we do at Credible.

You might start with a few tables in your primary database. Soon you may create a separate web app with a nightly cron job to sync data. Before you know it, you have more data than you can handle, jobs are taking way too long, and you’re being asked to integrate data from more sources. This is where a data warehouse comes in handy. It allows your team to store and query terabytes or even petabytes of data from many sources without writing a bunch of custom code.

In the past, only big companies like Amazon had data warehouses because they were expensive, hard to setup, and time-consuming to maintain. With AWS Redshift and Ruby, we’ll show you how to setup your own simple, inexpensive, and scalable data warehouse. We’ll provide sample code that will show you to how to extract, transform, and load (ETL) data into Redshift as well as how to access the data from a Rails app.

I created a tutorial for Redshift and Ruby here and wrote about it on our new engineering blog.

Why VR is Ready but Robotics and Other Tech Aren’t


When I talk about Virtual Reality (VR), my heart rate increases, my volume climbs, and before you know it, someone’s asking me chill-the-frak-out. Oftentimes people who haven’t seen the light scoff at the notion that VR is ready for take-off. In fact, they usually start touting something else. Timing is paramount in tech with the biggest example being Webvan, which was a spectacular failure, the “biggest dot-com flop in history“. However, the idea of online grocery delivery is now a booming business (e.g. Instacart, Safeway and Costco deliveries)–it wasn’t ready back then. Here are some tech trends and why they’re not ready to take off.


I’d consider any autonomous robot like the RoombaJibo or those creepy animal-bots part of this trend. Google’s self-driving car counts and quad/hexa/octo-copers kinda count as well. Unlike web apps, which just require software, robots need both hardware and software. You’ll need a robot base like the Roomba dev kit, some computer like Raspberry Pi, and then ideally some sensors and actuators (e.g. a camera and an arm so it can fetch the beer from the fridge for me). You can leverage commercial platforms like Sphero, but these have few sensors and thus aren’t very interesting (I’ve done some ruby programming with it). Until hardware costs go down, you’ll be hard-pressed to find people writing software for it. For example, a big reason why Google Glass didn’t take off was because it cost a whopping $1500 and wasn’t even 3D (the other reason was that people who had them tended to be Glassholes). Even when the hardware becomes dirt-cheap, writing software for interacting with people and the real world is fraking tough. We have a hard enough time getting websites and mobile apps to not suck today.

Internet of Things (IoT), EdTech, and BioTech

With platforms like SmartThings and Amazon Echo, you can do awesome things like turn off your lights by saying “Alexa, turn off the living room lights”. Echo costs $180 and smart light bulbs costs at least $30–10x the cost of a normal light bulb (how many light bulbs do you have in your house?). IoT also requires good WiFi usually and we all know how good that is. IoT devices are also much more useful if they’re portable but we have the issue of battery life, which we’re familiar with because nobody’s phone lasts more than a day. As for the software, Amazon Echo, Microsoft Cortana, Apple Siri, Google Now, are all in their infancy. Heck, most of the Google Now team just quit. Education and Healthcare technology are mired in regulation and bureaucracy. That’s a common cause of tech trends taking longer than expected. Quads and self-driving cars are starting to see the same hurdles appear too. Take a look at the FAA Drone Regulations or the self-driving car laws that were introduced across the US in 2015.

VR and the Order of Things

The Oculus Rift DK2 costs $350. The first consumer version of the Rift will be ~$300-$500 and $1500 if you include a gaming PC. Keep in mind that for most of the gaming community will already have the necessary PC hardware. Samsung’s GearVR is $200 if you own one of their high-end phones and all the way down the line, you can get a Google Cardboard for less than $10. The Oculus Rift will soon be sold as a consumer product at scale in the price range of other, more mature products like Xbox One ($350), Playstation 4 ($400), iPhone ($650 off-contract), and iPad Air 2 ($500). Given that Oculus has already sold 175,000+ dev kits, we’ll be seeing VR products under Christmas trees for years to come.

For software, VR leverages the game industry. Popular game engines, the foundation of VR software, are either freemium (Unity3D, $1500 pro version) or have no upfront cost (Unreal Engine, 5% royalty). Microsoft Visual Studio, the standard IDE, is also freemium. With these tools, you can already train to be a Jedi (I’ve tried this and it’s even more awesome than it looks). Granted you need expertise to create immersive 3D worlds but much of the core technology like physics simulation and dynamic lighting are already built-in. Furthermore, with new techniques like Blueprints (Visual Scripting) for Unreal Engine allowing developers to write less if any code, it will become easier and easier to build for VR.

Unlike other tech trends, VR doesn’t have high hardware costs and an immature developer ecosystem. There are also few if any legal or ethical hurdles holding it back. VR has a solid base within the games industry, as evidenced by the recent launch of Youtube Gaming where gamers playing VR demos feature prominently on the homepage. Once the commercial hardware starts selling next year, more developers will write software for it. With more software available, more people will buy the hardware.

Tech trends don’t happen in isolation and I believe there’s a rational order of things. VR will help us simulate and test robots, drones, and anything else with high hardware costs cheaply and quickly. VR will help us harden software for self-driving cars so that lessons are learned through virtual accidents rather than real tragedies. VR changes how we develop technologies forever: software allows us to work with bits, VR allows us to work with atoms.