It’s the small things

At 15 crosswords per page and 129 pages, teachers have generated almost 2000 crossword puzzles (it’s been about a month). It’s one of life’s small things, but it’s a nice feeling–that something you built is useful.

Project Management for Startups

In large companies, project management is a must; in startups, it’s a radioactive “hot” potato. Large companies have dedicated project managers and sometimes even a Project Management Office (PMO). Small companies like ours don’t have one and we’re pushing 60 employees now after 18 months. Eventually, it’ll make sense to have a full-time project manager, but until then we want to stay agile so we get by using software, process, and face time.

Use the Right Software

When our Product-Engineering team consisted of CEO, CTO, and two engineers, email was our issue tracker and a whiteboard contained our roadmap. That worked well. Soon we hired a few more engineers and email proliferated. Pivotal Tracker (PT) became our issue tracker (still free then!). PT is simple, light, and an engineering team’s dream–a place where only engineers could hang out. We upgraded to Google Spreadsheets for our roadmap so we can do more sharing and simple Gantt charts. Once we hired some product managers, added a QA team, and started developing many projects at once, it was time to swim in the Olympic-size pool: Jira. Besides being created by a hilarious bunch, Jira does everything. That’s part of the reason we resisted using it early on, since we didn’t want to fill out so many corporate-y fields like “hours worked” (::shudder::). As for our roadmap, it broke Google Spreadsheets and I grew tired of the “aw, snap!” pages. I’m experimenting with a few tools and so far I like Asana and Smartsheet. Find what works for you now.

Evolve Your Process But It’s all About Face Time

We tried to add only as much process as we needed over time. You start out with a fast track from feature conception to release. When you have two people who work on a project, you can sit down and have a heart to heart, nod heads, and then start coding. When a product manager has to convey thoughts to a few developers and a few QA–this is a small team still–details get lost in translation. Not only that, conversations may happen between members of the team that aren’t shared with the rest of the team. Let the frustration and angst begin. For a while, we went with the Product Requirements Document (PRD) approach, which is very monolithic. We’ve since moved away from that. Recurring meetings get moved around and their titles and agendas change. We went from 1-week to 2-week releases. Processes should change when the team changes or when the needs of the team change. It all comes down to getting a bunch of people to share ideas and work together efficiently. All this software and process is meant to cheat time–time to talk to one another face-to-face that is very hard to come by.


How to Find a Technical Co-Founder

Finding a technical co-founder is like online dating: too many guys and not enough women, except you skip the engagement and jump straight to the wedding. Often times, what’s really being asked is “How do I learn about starting a company?”. Check out the title of this hugely popular Quora question: “I am a creative guy with a startup idea. Where is the best place to find a rockstar developer to bring it to life?” There are 35 answers and it has been viewed 19870 times. The wording of this question reveals a troubling conceit–the idea that once you have an idea, all you need is to hire a few monkeys to code it up–then profit! The world is richer and more complex than that, my friends.

1 is the Loneliest Number

Apple had Steve Wozniak, Google had both Larry Page and Sergei Brin, Facebook had Mark Zuckerberg. It’s hard to build successful technology companies without a strong technical co-founder. You really need the wide range of talents from a Marketer, Product Manager, and Designer–it just so happens that the 1+1 combo of a business founder and technical founder can condense this set of skills into two people. With freelancers and advisers, you can keep the team size to two longer, but you’ll be hard-pressed to find a single person that does it all.

Be Realistic

Be honest with yourself first so you can be honest with others. Rate yourself on a scale of 1-10 for this role. Figure out what strengths you bring to the table and what’s lacking. Are you ready to be part of a 1+1? Most likely, this means you’ll be solely responsible for the business side of things including marketing, sales, legal, etc. Should you find a co-founder or should you join a startup to learn some skills first? If you’re not a superstar business guy, you will not get a superstar technical guy to work with you, so have realistic expectations.

Make Friends

Highly sought-after women generally don’t need online dating. A great engineer will have six-figure offers from Google and Facebook on the table. For them to pass on those jobs is like ditching a millionaire for a struggling writer. You better be Ernest Frakking Hemingway carrying a dead lion you just killed with your fountain pen. I suggest you make friends with as many technical people as you can. Go to meetups, conferences, Startup Weekend, etc. Make connections on Twitter and through your extended network. Meet your neighbors who may be trying to build the next Instagram (engineers tend to tinker too much and ideate too little). At some point you’ll realize why the term “rockstar developer” is so passe. If you need some technical advice, introduce yourself on Twitter–I go by @mankindforward.

Why your engineering team needs Vagrant

One of the first things I told our CEO when I joined as the first engineering hire was to buy every engineer a 15″ Macbook Pro. I wanted to have a consistent development environment for my team. I created a Google Doc and at first I ran through this How-To personally on every new machine for every new hire. The last 3 or 4 hires were given this doc on their first day and their setup time became part of the “How quickly can I setup my dev box” game. The record is 4 hours but he had most of the dependencies setup. The longest was two days.

It’s All Fun and Games Until Someone Upgrades

In the last 18 months, Apple has changed the processors, memory, and countless other aspects of the 15″ Macbook Pro. OSX has gone from Snow Leopard to Lion to Mountain Lion. Ruby has gone from 1.8.7 to 1.9.3, Rails has gone from 2.3 to 3.1, and I can’t keep track of all the versions of gems that have changed. The development world–it changes fast. How bad could it be, really? Well, I include 4 different manual “patches” in that doc of mine to make mysql work. The working patch depends on whether you have 32-bit or 64-bit OSX, your version of Xcode and mysql, whether you used a tarball or a dmg, whether you use mysql or mysql2 gem, etc. Lately, I just tell new hires to try everything until they find something that works. Even for packages installed via macports, certain version of ports don’t install properly. Some of these can be fixed by editing the portfile with information culled from the Interwebs. Other times, people just give up and use the homebrew version. This is us at 10 engineers.

Remove Doubt

Development and debugging can often lead to situations where you try to find what is different about a working scenario and a broken one. If a test passes on my machine but not on yours, I don’t want to say “it could be because you’re using Rails 2.3 vs 3.2 or Ruby 1.8.7 vs 1.9.2” or god-forbid “it could be because you’re using Windows”. Ideally, I would like to say “it’s because you forgot to pull the latest code” or  “your custom configuration file is not consistent with mine”. The more differences between one environment and another means the more variables to consider–follow the rabbit down the hole.

Abstraction FTW

Just as rvm helps engineers manage Ruby versions and bundler, Ruby gems, Vagrant helps engineers manage their entire development environment. With the concept of “boxes”, an engineering team can build Virtual Machine images to suit their every need. Want to onboard a new hire? Give then a computer and have them load your team’s starter box in 5 minutes. Want to roll out security updates to all 500 of your engineers? Script everything up using Chef or Puppet and have everyone download a new box. Want to test how your app would run in Ubuntu rather than Gentoo or OSX? I just did that while writing this post. When do I really need vagrant? For small teams (2-3), you’ll probably live without Vagrant. However, if you want peace of mind or plan to scale, check it out now.

Every Startup Needs a Gong

This is our third gong. Our first one was a few inches in diameter and sat on a desk. Our second one is about six inches in diameter–a mini version of this beast, which vibrates your soul if you stand too close. Gongs are a meme at Badgeville–a unique story about our company just like “Team Punishment” and Iceland. We hit the gong whenever sales closes a deal and the size of the deal determines the number of gongs (and now also what size gong you hit). Other companies also does this for sales. I hear New Relic gongs and sends a mass email to the entire company when a client goes premium (awkward and funny story around that). I love the gongs because they’re a tangible reminder of progress. As an engineer living in a world of 0-downtime releases and in the past weekly or even daily releases, there’s no party to commemorate the mailing of CDs with your software. Somewhere out there, some VMs turn over in a dark, temperature-controlled room and your SaaS product has been updated.

Gongs are one of the many ways companies develop culture and character. Our name is “Team Punishment”. The origin of the name is oh-so-random but it’s stuck nonetheless. Every quarter we design t-shirts to commemorate the evolution of “Team Punishment”–recent ones featured images from the Godfather and Scarface. I remember one quarter we had one w/ a japanese phrase translated as “team with no worthy enemies”. As we’ve grown to over 50 employees and moved 4 offices and now occupy a fifth across the street, it’s the little things like gongs and team names and backstories that pass on the culture, the essence of the company. The space-time continuum in a startup is compressed–years feel like months and weeks feel like days. It’s very easy to get caught up in Getting Stuff Done (GSD), but it’s these tiny details that help tell the story of your company.

Why Google is Dying to be More Social

It’s tiring to hear that Google doesn’t have “social” in its DNA. I left a comment on the article, but I must elaborate on this battle of epic proportions happening in the tech world.

Arrogance is a Ruse

Google is or can be perceived as being arrogant. It’s unlikely that a company that has transformed the world as Google has can avoid a tinge of arrogance. Hence, it is naive to think that Google’s attempts at “social”, namely Google+, are merely whimsical dalliances of an aging giant. I don’t think anyone at Shoreline realistically thinks Google+ will ever overtake Facebook. The arrogance is a ruse to throw us off the scent–the smell of deepening fear.

Beyond PageRank and Wide Open Spaces

Google’s stated mission is to “organize the world’s information and make it universally accessible and useful”. When what all the world knew was Yahoo’s labyrinthian category hierarchy, Google’s algorithm for indexing and searching the growing sea of information was a quite a paradigm shift. This approach is still dominant today but the sea itself is changing. The Internet used to give Google free reign to crawl and index. Today, with digital fortresses like social networks and pay walls, it’s become more and more difficult for Google to complete its mission. For a while, Google had a deal to incorporate tweets into their search results. As of late, the relationship is still strained. Then there was the whole Google-Facebook address book debacle. More and more, the world’s information has evolved from merely flat web pages to intricate graphs containing not only people but brands, topics, and even pets. Companies like Badgeville and lately Facebook as well, go further, building “behavior graphs” where the connections between the content are rich verbs like “watched”, “purchased”, and “performed”.

Social or Bust

Google’s frustration is apparent and with good reason. Think back to Google’s ordeal with the Chinese government last year. The situation is different, but the problem is the same: the world is not as open as Google likes. The government writes the rules in China and Google must operate by them. It took Google a while to realize that. The same battle rages on between Android and iPhone. For better or worse, Apple keeps tight control over its iWorld. Some people like that approach, but for the rest of the market, Android is bringing order to the chaos. Google+ is an attempt to break down yet another set of barriers in another arena. It’s the counterweight to Facebook and its unique way of looking at the world. Whether or not you believe what Google believes, you should respect them for sticking to the mission, even if they don’t always clearly articulate them. However, you really have to wonder how Google will organize the world’s information when it has no access to large portions of it. Or perhaps what will happen if Google stopped fighting these fights.

How to Use MongoDB with Engineyard

We have scaled our primary database, MongoDB, on Engineyard over the last year. We started with a single mongod process running version 1.6.4, then went to 1.8.1 and now at the latest stable version. We’ve used master-slave, replica set, and also replica set + sharding. We’ve learned a lot about how to deploy MongoDB on Engineyard and wanted to share that with the startup community. I’ve even heard that Engineyard is working on productizing some of the configuration so you can launch a MongoDB just as easily as you use MySQL. While that’s still in the works, here are some specific tips from our experience that may save you time.

Getting Started

A good place to start is Engineyard’s best practices for MongoDB and custom Chef recipes. You should also familiarize yourself with the MongoDB documentation and perhaps even check out the Jira project that is viewable to the public. We’ve found the documentation for Mongo through and up-to-date. We deployed our first MongoDB environment using the example recipe. I suggest playing around with that for a while to get comfortable–e.g. mongo console, tutorials and such. Once you’re comfortable with a simple replica set, you can move on to more advanced recipes for sharding only if you need it. The transition isn’t too bad because a sharded environment is composed of individual replica sets.

Take Advantage of the Flexibility

We run multiple configurations in different environments because it’s expensive to shard every environment or even have replica sets. In your staging or developent environments, you probably only need a single MongoDB instance. The good news is that your application configuration can easily accomodate different these different configuration. For a sharded environment, you’re connecting to the cluster through “mongos”–this means your cluster looks like a single mongod process to your app. Having good recipes is the key to have everything work as they should. Develop some recipes for your app using the example, Engineyard, and other people (like us!) to customize the configuration to your needs. For example, we specify the version of MongoDB we use on every environment to have absolute control.

Know Your ORM and Drivers

When your recipes are good to go, you don’t need to change them much if at all over time. Getting your application working well with MongoDB can be a challenge. Recently, there have been big transitions in this world–Rails 2.x vs. 3.x, Mongoid 2.3.x vs. 2.2.x, BSON, BSON-ext mongo-ruby-driver 1.3.x vs 1.4.x vs 1.5.x There can be conflicts with Rails and Mongoid if you’re not careful about where you are on the Rails 3 divide. Drivers had issues going from 1.3.x to 1.5.x since 1.4.x was release and then quickly yanked from rubygems. On top of that, MongoDB was transitioning from 1.8.x to 2.0.x, which was a big change. The chaos will subside but knowing where all the pieces lie for you is crucial to avoiding problems. A configuration that worked for us was:

  • Rails 3.1.x
  • Mongoid 2.4.x
  • mongo-ruby-driver, bson, bson-ext 1.5.x
  • MongoDB 2.0.x
I recommend Engineyard and 10gen support to help with big deployment. We pay both for premium support but there are plenty of documentation resources and free support available if you’re strapped for cash. Of course, you can always find help on twitter along with @MongoQuestion as well as quora.

Special thanks for Ines Sombra, MongoDB expert on the Engineyard Data Team, who has helped us tremendously over the last year!

MongoDB + Hadoop = gg

I’m not sure if kids these days still do this, but when I was a gamer and I had just beat someone soundly, I simply typed “gg”. It’s short for “good game” but it really means “I just dominated you” or “game over, loser”. When you use Mongo with Hadoop, it’s kinda like that. With Mongo, you get a flexible, scalable database that excels at real-time processing. More and more startups are using it today and it’s our primary database (here’s why). With Hadoop, you get a distributed processing framework that handles everything you can’t or don’t want to process in real-time. It’s even easier to scale than Mongo, Amazon’s productized it (Elastic Map/Reduce), and it’s the swiss army knife of Big Data. Here are some reasons why they are a match made in data store heaven:

You Complement Me

Mongo is fast; it’s optimized for speed. Say goodbye to transactions and joins and other features you may not need. You can use it as your primary database to support your application in real-time. It’s not so good on large datasets or complex querying. You lost joins remember? And you want to shard my what? This is where Hadoop comes in. If you’re thinking about doing Big Data analytics, take those Nginx logs and crunch those numbers in your Hadoop Cluster. Or if you’re tinkering with the latest machine learning algorithms to predict your users’s preferences–Taste Graph anyone?–it comes in handy.

Two Words: Map and Reduce

Map/Reduce (M/R) is at the core of Hadoop. It allows you to break down complex tasks into manageable chunks of data and processing. Mongo took a page out of Hadoop’s book when it included an implementation of M/R. It makes it even easier, in my view, because writing the functions in Hadoop’s native Java is usually more confusing then writing Javascript for Mongo. Add some Ruby and you’ve got dynamic M/R in your Rails app! You can write mappers and reducers in Mongo to validate your Hadoop Java code on a smaller data set. And when you’re ready for the big show, you can fire up your 1000-server cluster to find the question to 42.

Have Your Cake and Eat It Too

If you don’t know what to choose for your task, you can always use both at the same time. With a plugin, you can use mongo as an input or output for Hadoop. It even has some optimizations for splitting the input on every chunk in a sharded environment. We’ve tried this for one of our features and it works very nicely. Eventually, if your data requirements may grow such that you’ll have to go fully into Hadoop, but you can get away with this hybrid approach for a long time. If you’re looking to speed up processing time, you could farm out some data to Hadoop, have your cluster crunch the data in bite-size chunks, and do some more processing in your application–all within a Resque job.

Art Credit

Why I Joined a Startup

There I was, working for Andrew Ng, known for his research in machine learning and computer vision. I was lead on Stanford AI Robot (STAIR), getting the project off the ground and teaching the robot to wheel around Gates and open doors with its robotic arm. Needless to say, I was very fortunate to have an interesting project. And yet something was missing. As Larry Smith said, “What you want is passion–it is beyond interest“. There I was, having a quarter-life crisis.

I don’t have an accent and though from time to time I talk like a fob or a banana for fun, my parents did a pretty good job raising me in the Chinese tradition balanced by America’s progressive values. In high school, my weekends were mostly spent studying for the SATs and when my parents weren’t looking, playing the original Quake on a 13-inch CRT. When I wasn’t in China visiting family during the summers at Yale, I worked at research labs on things like Molecular Electronics and the Autonomous Vehicles. I didn’t do internships in industry because that wasn’t in the plan. The plan was to get that Ph.D. because in Chinese culture, the degree is king. In a culture whose recent history rewarded political office to the best scholars and with it a life of fame and fortune, this mindset is hard to shake.

There I was, fulfilling the plan but not the passion. So I took baby steps. I worked on a few startup projects with friends and acquaintances and learned a bit of Ruby on Rails, which was in its toddlerhood (1.0, baby!). I wanted to build things that had impact right then–I knew I had at least a few decades to wait until robots would advance beyond glorified vacuums. I tried to read about the startup life but barely scratched the surface. To be honest, I didn’t really prepare myself properly. I had the comfort of graduate school to cushion me while I did some part-time work for a company called Howcast. One day I was working from home part-time and the next day I was “WFH” full-time. I still played basketball with my grad school buddies after work. There I was, a newly minted startup guy.

There are many reasons not to join a startup so back then I listed many of them to myself. People with young children or other family responsibilities. Nope, I left that box unchecked. People who can’t handle high-stress or high-intensity situations. Nope, my personality craves andrenaline–like playing FPS for 16 hours straight w/o a bathroom break or popping a few Lactaid pills before doing the Milk Challenge. Ultimately, this was a lame game to play and since “everyone was doing it”, I definitely got sucked in by the hype as well.

Here I am, a startup guy. I can tell you that you’ll learn so much more and have so much more responsibility at a startup, but that may not always be true. I can tell you that the potential rewards are greater than the risks, but I won’t go on the record with that. I can tell you that you’ll have more impact than teaching or working at a big company but that’s not a truism either. When you’re in a quarter-life or mid-life or three-quarters-life crisis, it’s a “solution” you’re looking for. It’s the sign in the road that says “Happiness 42 miles”. Well, there were budget cuts and those signs were never installed. No path to follow, no one end result to optimize for. And guess what, startup life may or may not be for you. It’s like saying, “Surprise! You’re on the Truman Show!” My reasons have changed over time but one thing hasn’t really changed. Here I am, a guy who isn’t ever bored for long.