(2023-07-15) Maurya The Art Of Experiment Design

Ash Maurya: The Art of Experiment Design. Running experiments is a key activity for continuous innovation. But simply running a bunch of experiments is not enough.

Why? Because a hypothesis is a guess. And, the quality of your results directly depends on the quality of your guesses.

How do you formulate better guesses? That’s the topic of today’s issue.

1. Adopt a Discovery Before Experiments Mindset

for any given problem, it’s pretty easy to guess a multitude of possible solutions.

We often prioritize solutions that align with our pre-existing worldviews (or biases).

A better way is holding off solution prioritization until you better understand the underlying problem, i.e., going beyond surface problems to root causes. (problem discovery)

2. Shortlist Your Most Promising Validation Campaign(s)

3. Test Your Most Promising Validation Campaign Using Small, Fast Additive Experiments

I recommend using 2-week sprints.

This is when you formulate falsifiable hypotheses. Examples: At least 100 people/week will start watching the first product walkthrough video (Acquisition)

The power of using small timeboxes is that you get faster feedback on the overall campaign. If enough people are not finding or watching the first video, adding more videos will not help!

Good validation campaigns should start with problems before solutions or discovery (D) before traction (T)

When designing any campaign, it helps to consider these seven questions:
Discovery: Is there an underlying problem worth solving?
Acquisition: Are enough people interested/impacted?
Activation: Does it deliver value?