Why I Started Frank

Upom Malik • May 5

The Life of a Company Experimentalist

"Can we test a third mobile layout?"

"Well...we could, but we would reduce the minimum detectable effect we could measure."

The product manager looked at me with some cross of annoyance and loss. I couldn't blame him, though. It's a look I had seen time and time again when working with product folks to run experiments. Sample size calculations, setting feature flags, interpreting confidence intervals. It was all a slow and frustrating process.

Product folks were not the only source of complaints. The data scientists and analysts on my team also had their share of gripes. The folks I worked with are incredibly bright, and there were few things they loathed as much as running the same 10 calculations over and over again for our product experiments. It was a rare week where a 1-on-1 with my direct reports did not involve a bored data person who felt disengaged with a full-time job of babysitting experiments.

This was the way things ran day-after-day at work. And somewhere in the back of my head, the same thought rang: there has to be a better way.

Why Experiment?

Experimentation lies at the heart of all great innovations. Only by trying new things can a company eventually make new discoveries that drives it to success in the marketplace. The recognition of this fact has led many companies to invest in and build cultures of experimentation. Companies like Google and Airbnb run thousands of experiments each year, allowing them to both systematically find insights and produce better customer experiences. In an oft-cited example of the power of experimentation, Amazon engineer Greg Linden ran an early test of Amazon's recommendation systems against the wishes of a senior executive, and conclusively showed not using recommendation systems was costing the company a huge amount of revenue. The vast majority of companies can similarly benefit from experimentation in their business.

The Problem

Unfortunately for all but the largest tech companies, running experiments is anything but easy. In most companies, the entire process of running experiments involves an amalgamated mess of infrastructure, DIY tooling, and manual process that makes it incredibly difficult to setup and run an experiment. Because of the often technical and quantitative work involved, such experiments end up requiring a specialized data team to execute, even though much of this work is incredibly tedious and ripe for automation. Large tech companies have for the most part escaped this experiment hellscape by investing significant resources in building self-service experiment platforms that make it easy to run experiments, and ultimately scale up their experimentation efforts. This stark difference in experimentation setups and cultures between large and small companies has been referred to by one observer as an experimentation gap.

However beyond just getting an experiment set up and analyzed, much of the tooling currently available is outdated for modern company needs. Over the last few decades, there have been many advances in statistical inferencing, data processing, machine learning, and artificial intelligence that not only allow insights to be pulled out easier, but also allow dynamic optimization on the fly. Yet most modern experimental tooling still utilizes only simple frequentist A/B testing, and foregoes many of these new capabilities.

...And a Solution

It is for this reason I decided to start Frank. All companies should have access to experimentation tools that make it easy as possible to try something new, gain insights from it, and apply those learnings to improve outcomes. If the contents of this article resonate with you, then sign up for our mailing list, or feel free to reach out to me on Linkedin. I'm looking forward to hearing all about your experimentation challenges, and how we can work together to solve them.