Liars Fear Data

Screen shot 2013-02-17 at 11.12.09 AM

 

There was a fierce battle of words this week between the New York Times and Elon Musk, the CEO of Tesla. As the story goes, the NYT published a review of the Tesla S that trashed the battery performance and charging system. Elon thought this was suspicious since most other reviewers found it awesome, and Motor Trend says it’s the 2013 Car of the Year. He did a thorough analysis of the log data (whoops, reviewer probably didn’t think of that) and fired back some accusations in a blog post on the Tesla web site. The NYT reviewer fired back with some excuses and by that time the issue exploded. The back story is that Elon had been burned before by the media. A few years ago, Top Gear did a show on the Tesla Roadster and scripted up a dramatic episode where the Roadster’s battery died after 55 miles (it didn’t). Elon sued and lost, since it’s an entertainment show. This time, the usually measured CEO and Tony Stark character model went above and beyond the call of duty for his pride and joy.

Reactions among people vary. Some say Elon should have let the data speak for itself. Some say he should have put it behind him quietly. I think he did the right thing for himself and by his company. If it were a less well-known media source, I’d recommend playing it cool. If it were a less obvious twisting of facts, I’d tell him to let others fight the battle. However, this was the New York Times and this was after many other reputable news sources had given very positive reviews and this was a “fool me once shame on me…” situation. As a leader, Elon knew when to play it cool and when to pull out the big guns. If he had let this go, there would be Doubt in the market. NYT’s word vs Tesla–obvious NYT has more sway today. If he had let this go, he would have invited many other challengers and haters to take shots at him. Maybe this type of sleazy review happened before but this is the first he’s rebutted? If he had this go, he would have let not only his investors down but himself, and more importantly, his employees. If you’ve seen “Revenge of the Electric Car“, you’ll understand that it’s been a long and tough road for his company. Elon’s been fighting all the way to get here and just when the company is about to be mainstream, this liar at the NYT decides to get some page views at Tesla’s expense. I bet Elon must have said:

I

don’t

think

so

*******.

Balance of Power in #startups

Startups are like small countries–if the balance of power is not maintained, bad things like death and destruction happen. The following is a set of scenarios you want to avoid.

All Hail Engineers

The technical half of a two co-founder startup dominates all decisions. He starts out as the CTO but muscles his way into the CEO spot, leaving the business co-founder with a VP of Sales position. The company plans to hire 50 new engineers and a minimal sales staff because the CEO believes the product will “sell itself”. Support is dying for more resources, marketing is dying for more resources, but engineers are paid way over market. This is reminiscent of Facebook in its early days and you’ll find many stories of talented people leaving the company because they were unhappy with the fact that the culture was too engineering-centric. Remember the movie “Social Network” by Aaron Sorkin? Avoid that.

Rowing the Boat

The business half of a two co-founder startup dominates all decisions. He is CEO and there is no CTO, just a bunch of engineers. The company is very sales-focused and eschews things like R&D. Engineering is viewed not as a source of innovation but a team of slaves “rowing the boat”. This makes me think of today’s Yahoo. The talented research team was fired, many of the engineers have jumped ship, and one of its multiple personalities thinks of itself as a content company. Yahoo isn’t a startup anymore, but there’s a strong tendency to forget–especially as startups grow up–that in the tech world you innovate or someone will innovate you out of existence. Without a strong engineering voice, you might as well be walking the plank.

Microcosms

The first two scenarios are the most glaringly obvious ones that are easy to spot–others take a trained eye. When a startup hires a stable of VPs and each one is off to the races on Day 1, it’s hard to have them all run at the same pace. Some departments may not even have a horse in the race! It’s subtle, but you will be able the see the effects of unbalanced power. Instead of quick and decisive course corrections, you’ll get reactionary jerks. For example, when Netflix announced that they were raising prices and killing off their DVD service in a cold email, I suspect something was awry in the top leadership because up ’til then, Netflix was known for its customer relations. They refunded customers when service outages happened, CEO Reed Hastings routinely wrote candid emails to customers explaining decisions, employees broadcasted their uber-flexible vacation policy, etc. Certainly, a balanced leadership would not have supported such an uncharacteristic move?

Maintaining Balance

This is damn hard for a startup. You start with 1+1–that’s easy. That’s a simple handshake. You add a VP or two after you get your Series A–not too bad. That’s a basketball team huddle. You add a few more to grow your other departments after you get your Series B–hm… That’s a football team huddle. You add SVPs and let’s not forget the ever-growing Board of Directors after your Series C–shiznit. That’s a village. It’s a challenge, but the company has to listen and weigh equally, every single voice all along the way, even if some are quieter than others and especially to those voices that are missing.

Project Management for Startups

In large companies, project management is a must; in startups, it’s a radioactive “hot” potato. Large companies have dedicated project managers and sometimes even a Project Management Office (PMO). Small companies like ours don’t have one and we’re pushing 60 employees now after 18 months. Eventually, it’ll make sense to have a full-time project manager, but until then we want to stay agile so we get by using software, process, and face time.

Use the Right Software

When our Product-Engineering team consisted of CEO, CTO, and two engineers, email was our issue tracker and a whiteboard contained our roadmap. That worked well. Soon we hired a few more engineers and email proliferated. Pivotal Tracker (PT) became our issue tracker (still free then!). PT is simple, light, and an engineering team’s dream–a place where only engineers could hang out. We upgraded to Google Spreadsheets for our roadmap so we can do more sharing and simple Gantt charts. Once we hired some product managers, added a QA team, and started developing many projects at once, it was time to swim in the Olympic-size pool: Jira. Besides being created by a hilarious bunch, Jira does everything. That’s part of the reason we resisted using it early on, since we didn’t want to fill out so many corporate-y fields like “hours worked” (::shudder::). As for our roadmap, it broke Google Spreadsheets and I grew tired of the “aw, snap!” pages. I’m experimenting with a few tools and so far I like Asana and Smartsheet. Find what works for you now.

Evolve Your Process But It’s all About Face Time

We tried to add only as much process as we needed over time. You start out with a fast track from feature conception to release. When you have two people who work on a project, you can sit down and have a heart to heart, nod heads, and then start coding. When a product manager has to convey thoughts to a few developers and a few QA–this is a small team still–details get lost in translation. Not only that, conversations may happen between members of the team that aren’t shared with the rest of the team. Let the frustration and angst begin. For a while, we went with the Product Requirements Document (PRD) approach, which is very monolithic. We’ve since moved away from that. Recurring meetings get moved around and their titles and agendas change. We went from 1-week to 2-week releases. Processes should change when the team changes or when the needs of the team change. It all comes down to getting a bunch of people to share ideas and work together efficiently. All this software and process is meant to cheat time–time to talk to one another face-to-face that is very hard to come by.

 

How to Find a Technical Co-Founder

Finding a technical co-founder is like online dating: too many guys and not enough women, except you skip the engagement and jump straight to the wedding. Often times, what’s really being asked is “How do I learn about starting a company?”. Check out the title of this hugely popular Quora question: “I am a creative guy with a startup idea. Where is the best place to find a rockstar developer to bring it to life?” There are 35 answers and it has been viewed 19870 times. The wording of this question reveals a troubling conceit–the idea that once you have an idea, all you need is to hire a few monkeys to code it up–then profit! The world is richer and more complex than that, my friends.

1 is the Loneliest Number

Apple had Steve Wozniak, Google had both Larry Page and Sergei Brin, Facebook had Mark Zuckerberg. It’s hard to build successful technology companies without a strong technical co-founder. You really need the wide range of talents from a Marketer, Product Manager, and Designer–it just so happens that the 1+1 combo of a business founder and technical founder can condense this set of skills into two people. With freelancers and advisers, you can keep the team size to two longer, but you’ll be hard-pressed to find a single person that does it all.

Be Realistic

Be honest with yourself first so you can be honest with others. Rate yourself on a scale of 1-10 for this role. Figure out what strengths you bring to the table and what’s lacking. Are you ready to be part of a 1+1? Most likely, this means you’ll be solely responsible for the business side of things including marketing, sales, legal, etc. Should you find a co-founder or should you join a startup to learn some skills first? If you’re not a superstar business guy, you will not get a superstar technical guy to work with you, so have realistic expectations.

Make Friends

Highly sought-after women generally don’t need online dating. A great engineer will have six-figure offers from Google and Facebook on the table. For them to pass on those jobs is like ditching a millionaire for a struggling writer. You better be Ernest Frakking Hemingway carrying a dead lion you just killed with your fountain pen. I suggest you make friends with as many technical people as you can. Go to meetups, conferences, Startup Weekend, etc. Make connections on Twitter and through your extended network. Meet your neighbors who may be trying to build the next Instagram (engineers tend to tinker too much and ideate too little). At some point you’ll realize why the term “rockstar developer” is so passe. If you need some technical advice, introduce yourself on Twitter–I go by @mankindforward.

Every Startup Needs a Gong

This is our third gong. Our first one was a few inches in diameter and sat on a desk. Our second one is about six inches in diameter–a mini version of this beast, which vibrates your soul if you stand too close. Gongs are a meme at Badgeville–a unique story about our company just like “Team Punishment” and Iceland. We hit the gong whenever sales closes a deal and the size of the deal determines the number of gongs (and now also what size gong you hit). Other companies also does this for sales. I hear New Relic gongs and sends a mass email to the entire company when a client goes premium (awkward and funny story around that). I love the gongs because they’re a tangible reminder of progress. As an engineer living in a world of 0-downtime releases and in the past weekly or even daily releases, there’s no party to commemorate the mailing of CDs with your software. Somewhere out there, some VMs turn over in a dark, temperature-controlled room and your SaaS product has been updated.

Gongs are one of the many ways companies develop culture and character. Our name is “Team Punishment”. The origin of the name is oh-so-random but it’s stuck nonetheless. Every quarter we design t-shirts to commemorate the evolution of “Team Punishment”–recent ones featured images from the Godfather and Scarface. I remember one quarter we had one w/ a japanese phrase translated as “team with no worthy enemies”. As we’ve grown to over 50 employees and moved 4 offices and now occupy a fifth across the street, it’s the little things like gongs and team names and backstories that pass on the culture, the essence of the company. The space-time continuum in a startup is compressed–years feel like months and weeks feel like days. It’s very easy to get caught up in Getting Stuff Done (GSD), but it’s these tiny details that help tell the story of your company.

Why I Joined a Startup

There I was, working for Andrew Ng, known for his research in machine learning and computer vision. I was lead on Stanford AI Robot (STAIR), getting the project off the ground and teaching the robot to wheel around Gates and open doors with its robotic arm. Needless to say, I was very fortunate to have an interesting project. And yet something was missing. As Larry Smith said, “What you want is passion–it is beyond interest“. There I was, having a quarter-life crisis.

I don’t have an accent and though from time to time I talk like a fob or a banana for fun, my parents did a pretty good job raising me in the Chinese tradition balanced by America’s progressive values. In high school, my weekends were mostly spent studying for the SATs and when my parents weren’t looking, playing the original Quake on a 13-inch CRT. When I wasn’t in China visiting family during the summers at Yale, I worked at research labs on things like Molecular Electronics and the Autonomous Vehicles. I didn’t do internships in industry because that wasn’t in the plan. The plan was to get that Ph.D. because in Chinese culture, the degree is king. In a culture whose recent history rewarded political office to the best scholars and with it a life of fame and fortune, this mindset is hard to shake.

There I was, fulfilling the plan but not the passion. So I took baby steps. I worked on a few startup projects with friends and acquaintances and learned a bit of Ruby on Rails, which was in its toddlerhood (1.0, baby!). I wanted to build things that had impact right then–I knew I had at least a few decades to wait until robots would advance beyond glorified vacuums. I tried to read about the startup life but barely scratched the surface. To be honest, I didn’t really prepare myself properly. I had the comfort of graduate school to cushion me while I did some part-time work for a company called Howcast. One day I was working from home part-time and the next day I was “WFH” full-time. I still played basketball with my grad school buddies after work. There I was, a newly minted startup guy.

There are many reasons not to join a startup so back then I listed many of them to myself. People with young children or other family responsibilities. Nope, I left that box unchecked. People who can’t handle high-stress or high-intensity situations. Nope, my personality craves andrenaline–like playing FPS for 16 hours straight w/o a bathroom break or popping a few Lactaid pills before doing the Milk Challenge. Ultimately, this was a lame game to play and since “everyone was doing it”, I definitely got sucked in by the hype as well.

Here I am, a startup guy. I can tell you that you’ll learn so much more and have so much more responsibility at a startup, but that may not always be true. I can tell you that the potential rewards are greater than the risks, but I won’t go on the record with that. I can tell you that you’ll have more impact than teaching or working at a big company but that’s not a truism either. When you’re in a quarter-life or mid-life or three-quarters-life crisis, it’s a “solution” you’re looking for. It’s the sign in the road that says “Happiness 42 miles”. Well, there were budget cuts and those signs were never installed. No path to follow, no one end result to optimize for. And guess what, startup life may or may not be for you. It’s like saying, “Surprise! You’re on the Truman Show!” My reasons have changed over time but one thing hasn’t really changed. Here I am, a guy who isn’t ever bored for long.

Engineyard or Heroku?

Web-based software is becoming more and more service-oriented. You probably know about software as a service (SaaS – e.g. Salesforce) and infrastructure as a service (IaaS – e.g. Amazon) but probaby not platform as a service (PaaS). Companies like Engineyard and Heroku make it easier to launch and scale a web application. For the most part, you can setup an account, run a few commands for basic configuration and you’ve launched in the Cloud. This is an order of magnitude easier than it was just 5 years ago. If you just felt your heart skip a beat, please take a few minutes to catch your breath. When you’re ready, let me help you decide what platform to launch your next application on.

Heroku is great for quick applications. The best Rails candidates I interview deploy their code challenges on there and send me a link. The smallest configuration on Heroku gives you a 5MB shared database and 1 “dyno”, their unit of measure for CPU. It’s free and you can even get add-on’s like New Relic performance monitoring thrown in. To scale up, you can pay more for dynos and a more powerful database. Setup is fairly painless, though I had to google for help when their “happy-path” documentation didn’t work exactly the way it was intended. The most annoying thing is that for the smallest configuration, the first request you make to the site takes 30-60 seconds to complete because their servers are dynamically provisioning you resources. Heroku hides much of the Operations side of things from you and that’s an intentional choice. It’s clear that they’d like to make the deployment process idiot-proof. I think that’s great for interview code challenges and school projects but if you’re serious about your application, you need to use Engineyard.

Why?

You need to be in control. Over the years I’ve seen EBS volumes become read-only all of a sudden, MongoDB refuse to resolve internal hostnames for a subset of instances, and even instances mysteriously become unreachable or “disappear”. Putting your servers in the Cloud has a huge drawback: you don’t have physical access to the hardware. When things go wrong–and they inevitably do–you need to have access. Since you’re saving money by not buying your own hardware up front and using IaaS, the next best thing is having as much control of your instance as possible. With Engineyard, you can ssh into every single machine and customize your own recipes. Essentially, they setup the default configuration but at the end of the day you can do whatever you want. For everything from Cron to Redis, I customize the configuration and understand exactly how processes are running. I analyze the CPU and memory usage and even the IO performance because I need to tune things like how many workers should be on each application instance. Control is critical when debugging issues. You’ll need to install custom monitoring and benchmarking to find out why your Memcached servers are maxing out at 80% memory utilization for example.

You need expertise. PaaS is your world-class Operations team. Even if you hired one or two Operations people, it is unlikely they will be experts in all the technologies you’re using. Given that I’ve never used paid Heroku Support, I can’t speak to their experts, but Engineyard definitely knows their stuff. They try to cover as many areas as possible in-house and through partnerships, cover others. I know that they have a great in-house MongoDB presence because their team helped me with my Sharding migration. I’ve gotten DBAs from Percona to look over my slow mysql queries, advice from Durran Jordan of Mongoid over IRC, and just as I was looking into Neo4j for some graph-based projects, I heard that they were actively talking with Neo Technology about a partnership. When your application has a problem and Google doesn’t help, you can either call a really smart friend like a contestant on “Who Wants to Be a Millionaire” or you can leverage the collective expertise of Engineyard and their network. They know your application and infrastructure really well and you may be seeing the same problem as another one of their clients. This expertise model works and just makes sense.

You need support. When you’re in the Cloud, you need someone to have your back. You can install a Heroku add-on like “Redis To Go”, but what happens when things break? As a engineer I’m pretty paranoid and rightfully so. I’m weary of things labeled “X to go” or “Y in a box”. Furthermore, I believe that if you’re going to use a technology in production, you should install it yourself. It’s critical that you develop a long-term relationship with a team that knows your specific application and infrastructure. When you read an email from Engineyard at 9am telling you how at 5am they detected a problem and brought your site back up without waking you from your precious 6 hours of sleep, you’ll know what I’m talking about. When it’s 3am and you’ve been wrestling with an issue that’s keeping your site down and your friends at Engineyard are still up and walking you through hell, you’ll know what I’m talking about. Heroku support may be just as good, I don’t know. But I’ve been in the trenches with these guys and I can tell you that whoever you choose, you better be able to count on them when the CEO is calling you to ask when the site will be back up.

If you’re serious about your application, you’ll take these points into consideration and weigh them heavily against cost and hype. Engineyard’s worth the money.
Disclosure: Badgeville is an Engineyard customer and so was Howcast when I was there