The Caveats of Being Metrics Driven

A/B testing has taken the technology sector and startup industry by storm, partially on the back of the phenomenon of “growth hacking”. Airbnb famously bootstrapped their first users by scraping Craigslist as well as allowing users an easy way to dual post back. Platform products that become more valuable with more users typically have a “chicken-and-egg” problem, and growth hacking is the key. Driving user numbers up and to the right exponentially increases the value of the service for all users.

I’ve used Groupon on and off over the years. It’s a cool place to find interesting things that you would not otherwise think of doing, like chocolate and cheese fondue meals, sky diving, and pottery classes. Let’s take a look at Groupon’s old home page. In the past, every time I want to check back to see if there is something interesting I get hit by this sign up prompt.¹

I’ve never worked at Groupon so I can only postulate as to how this happened. The more features and clickable links that were removed from the homepage, the more signups increased. Along the way Groupon probably hit all their OKRs/KPIs² while A/B testing this page into the ground. What is lost?

• Prospective employees who want to learn more about working at this company
• Business partners who want to investigate partnerships with this company
• Return customers who don’t want to fucking put in their email address again because it will sign them up to 15 different daily mailing distributions. Holy mother of Optimizely just take my money for the pottery classes without spamming me please.

Growth Hacking for companies that have cracked their chicken-and-egg problem is secondary to having a good product, and having good product is practically unmeasurable. Well, practical to measure but impractical for informing major decisions, because the one metric that correlates with a good product is long term retention. It will take months to measure and years to iterate on.

You do not have time to A/B test yourself into product market fit. If you are a startup, you don’t have time to A/B test at all. Spend your time building from your gut. The startup needs a strong product vision and a moral compass for really, really, caring about the user experience.

Here is Groupon’s latest homepage. It doesn’t look super sleak. Then and again it doesn’t have to (see: Facebook, Google). But look at all the things you can do here: window browse some offers, sign up your business, submit a resume.

Groupon is probably taking a short term hit on the number of emails signups they are getting, but their product is more functional now. In the long run, they are providing a better service and user experience. Sometimes the big picture involves not being too metrics driven. Good job Groupon. Go out and do good, unencumbered by the shackles of data!