Marketing experiment analysis & understanding your experiment results

Focus on learnings over results

Every experiment results in 1 of 3 outcomes:

1. Winner

2. Loser

3. Inconclusive

However, building a culture of experimentation isn't just about getting a success or fail. Every experiment is an opportunity to learn more about your customers and successful growth teams understand the more they learn about their customers, the more they will succeed.

“Every experiment you run should be designed so that, no matter the outcome, you learn something”

Chris Goward, GO Group Digital & WiderFunnel

It's OK To Fail

Failures aren't really failures at all. Experiment don't really fail as such at all, rather hypotheses are proven wrong - which is hugely valuable. Failing early saves time and money for your business and prevents time and money being spent on something that isn't important. It also enables growth teams to better understand changes that can have negative impacts on key objectives and KPIs.

Innovation and experimentation require taking risks, which means failure is to be expected. Being a successful growth marketer therefore, requires a level of grit, resilience, and perseverance. Failure isn't a failed initiative, failure is doing something and not learning from it.

What is important is that when an experiment fails, the results are analysed and learnings are shared. In many cases, failed experiments can deliver insights that end up having a much larger business impact than the initial successes.

All we have to do is to keep a high velocity of testing and test lots of ideas. The more we test the more value we generate. So instead of spending a lot of time trying to figure out which ideas to run, we run them all.

"Failure is simply the opportunity to begin again, this time more intelligently."

Henry Ford

What to consider during marketing experiment analysis

When performing marketing experiment analysis and writing up your results include what you measured and the test results, but also consider answering some of the questions below.

• Hypothesis: Has the hypothesis been proven or disproven? How close was it? Why did you get the result that you did?

• Accuracy: How accurate was the outcome? Were the results close or really far off? Why?

• Methodology: Was the implementation effective? What could/should you have done differently?

• Data: Did you gather adequate data, and interpret it correctly? Are the results statistically significant?

• External factors: Has something skewed the data, such as the time of year, or some promotional activity running concurrently.

• Future experiments: Should a new experiment be run to build on your findings? Is there a new hypothesis to test as a result?

• Anything else? Is there anything else you learned? Have the results affected how you think about your visitors/customers/users?

The most important question to continually ask during marketing experiment analysis is 'Why?'. This forces us to think about how our users, marketing channels or activities are reacting in a particular way.

“If only a quarter of experiments ‘win’, you have to figure out how to get value from the other 75 percent.”

Stephen Pavlovich, CEO of Conversion

Once your marketing experiment analysis is complete

Once your analysis is complete, and has been shared with your team, there are typically 3 options:

• Do nothing: Move on to other ideas in the prioritised queue that will likely have more impact.

• Iterate: Adjust with a slightly different hypothesis based on what you learned.

• Create a new hypothesis: Develop a new hypothesis entirely as a result of something you discovered.

• Double-down on success: Run the experiment again but with the goal of creating a bigger impact. Typically, this is where scope is expanded, along with resource, effort and/or budget. The confidence, potential impact and effort scores will all be much higher given this is an iteration of a previously successful test.

Creating playbooks and systems

Where possible successful experiments should be turned into repeatable playbooks or systems. These are easy to follow processes or checklists that form a knowledge base that can be used repeatedly, and improved on over time. This completes the last stages of the growth cycle.

HubSpot Growth Lifecycle

Why do marketing experiment analysis?

Digging into your experiment results, documenting and sharing the findings ensures that:

• Team learning: Your immediate team learns from the experience.

• Future leverage: Your team can use your results to fuel future ideas and experiments.

• Wider impact: Other teams can learn from your experience, potentially impacting other areas of the business.

• Experiment history: Your team builds a complete experiment history, effectively a library of customer insights, for future team members.

As a result, effective documentation is arguably the most important piece of the growth process.

Additional Resources

https://community.growthhackers.com/posts/how-to-run-experience-informed-experiments