Focus on learnings over results

Every experiment results in 1 of 3 outcomes:

1. Winner
2. Loser
3. Inconclusive

However, building a culture of experimentation isn’t just about getting a success or fail. Every experiment is an opportunity to learn more about your customers and successful growth teams understand the more they learn about their customers, the more they will succeed.

“Every experiment you run should be designed so that, no matter the outcome, you learn something”

Chris Goward, GO Group Digital & WiderFunnel

It’s OK To Fail

Failures aren’t really failures at all. Experiment don’t really fail as such at all, rather hypotheses are proven wrong – which is hugely valuable. Failing early saves time and money for your business and prevents time and money being spent on something that isn’t important. It also enables growth teams to better understand changes that can have negative impacts on key objectives and KPIs.

Innovation and experimentation require taking risks, which means failure is to be expected. Being a successful growth marketer therefore, requires a level of grit, resilience, and perseverance. Failure isn’t a failed initiative, failure is doing something and not learning from it.

What is important is that when an experiment fails, the results are analysed and learnings are shared. In many cases, failed experiments can deliver insights that end up having a much larger business impact than the initial successes.

All we have to do is to keep a high velocity of testing and test lots of ideas. The more we test the more value we generate. So instead of spending a lot of time trying to figure out which ideas to run, we run them all.

“Failure is simply the opportunity to being again, this time more intelligently.”

Henry Ford

What to consider during marketing experiment analysis

When performing marketing experiment analysis and writing up your results include what you measured and the test results, but also consider answering some of the questions below.

Factor	Question(s)
Hypothesis	Has the hypothesis been proven or disproven? How close was it? Why did you get the result that you did?
Accuracy	How accurate was the outcome? Were the results close or really far off? Why?
Methodology	Was the implementation effective? What could/should you have done differently?
Data	Did you gather adequate data, and interpret it correctly? Are the results statistically significant?
External factors	Has something skewed the data, such as the time of year, or some promotional activity running concurrently.
Future experiments	Should a new experiment be run to build on your findings? Is there a new hypothesis to test as a result?
Anything else?	Is there anything else you learned? Have the results affected how you think about your visitors/customers/users?

The most important question to continually ask during marketing experiment analysis is ‘Why?’. This forces us to think about how our users, marketing channels or activities are reacting in a particular way.

“If only a quarter of experiments ‘win’, you have to figure out how to get value from the other 75 percent.”

Stephen Pavlovich, CEO of Conversion

Once your marketing experiment analysis is complete

Once your analysis is complete, and has been shared with your team, there are typically 3 options:

1. Do nothing and move on to other ideas in the prioritised queue that will likely have more impact
2. Iterate with a slightly different hypothesis based on what you learned
3. Create a new hypothesis entirely as a result of something you discovered
4. Double-down on your success by running the experiment again but with the goal of creating a bigger impact. Typically, this is where scope is expanded, along with resource, effort and/or budget. The confidence, potential impact and effort scores will all be much higher given this is an iteration of a previously successful test.

Creating playbooks and systems

Where possible successful experiments should be turned into repeatable playbooks or systems. These are easy to follow processes or checklists that form a knowledge base that can be used repeatedly, and improved on over time. This completes the last stages of the growth cycle.

Why do marketing experiment analysis?

Digging into your experiment results, documenting and sharing the findings ensures that:

1. Your immediate team learns from the experience.
2. Your team can leverage your results to fuel future ideas and experiments.
3. Wider teams can learn from your experience (your results and insights can help impact other areas of the business in ways you may not realise).
4. Your team builds a complete experiment history, effectively a library of customer insights, for future team members.

As a result, effective documentation is arguably the most important piece of the growth process.

Additional Resources

https://community.growthhackers.com/posts/how-to-run-experience-informed-experiments