Table of contents
Many of the processes your company employs, especially strategy, roadmap, and project planning, are meant to create certainty — Here’s what we need to do and by when. The planning waterfall moves nicely from strategy to roadmap to projects to tasks.
But marketing is anything but certain. Everything is changing around you - your products and services, your customers, your ad platforms, search engines, new acquisition channels, best practices, technology, and your company.
Your well-crafted plans are likely entirely based on opinions. This is called false-certainty planning or false predictability, the classic 12-month roadmap being the most obvious example. You create plans or make decisions based on assumptions that are treated as certainties, even though they are inherently uncertain. This approach can lead to overconfidence in the plan’s success, resulting in an inability to adapt when conditions change or when initial assumptions prove incorrect.
Marketing experimentation is the practice of testing a marketing idea on a small scale, against a written hypothesis and a single metric, before you commit budget to it. You will also see it described as running marketing experiments, a marketing experiment (one test) or experiment marketing. It replaces big-bang campaign guesses with a steady loop of small, measurable bets.
How the Wright Brothers used agile & experimentation
In the late 1800s, governments around the world had invested over a billion dollars in aviation projects led by prominent scientific minds that had tried (and failed) to achieve powered man flight. Ultimately, they were beaten by two bicycle enthusiasts known as the Wright brothers.
But why did the Wright brothers succeed?
Instead of building an entire aircraft and trying to get it to fly, as many of their predecessors had done, the Wright brothers completed over 700 flights in gliders first. They started small, gathered data, and continuously iterated their way to success - building, measuring, and learning along the way. The timeline below summarizes the story of their success:
-
1899 - 1.5m wingspan kite
-
1900 - 5.3m wingspan tethered glider
-
1901 - 6.7m wingspan untethered glider
-
1902 - 9.8m untethered glider with rudder
-
1903 - 12.3m powered airplane, short straight-line flight
-
1904 - 12.3m powered airplane, short circular flight
-
1905 - 12.3m powered airplane, 30-minute flight duration
Experts believe it was this continuous learning and experimentation, what we would now call an agile approach, which led to their success.
The problem with campaigns & waterfall project delivery
Marketing projects typically follow the classic waterfall project delivery approach, and tend to look something like the following:
-
Come up with a theme for your marketing project or campaign
-
Seek approval from your manager and stakeholders
-
Build out the plan based on the agreed requirements, with support from in-house teams, agencies, designers, and developers
-
Bring together all campaign assets and test the user journey
-
Launch
Unfortunately, with this big bang approach, you take one huge swing and if you miss - if your target audience is wrong, the messaging doesn’t resonate, people don’t engage or convert - it’s over.
If the initial plan was wrong (and let’s face it, unless you’ve run exactly the same campaign before under very similar conditions, it’s likely more of a guess than a plan), you have just spent considerable time and money building a campaign that nobody pays attention to and that achieves nothing. Big bang = big risk.
The problem is that the waterfall project methodology was developed during the mass production era (car manufacturing, steel production, tobacco production, etc) where problems were well-defined and the solutions clearly understood.
Using waterfall project management you can launch the “perfect” marketing campaign - one that is on time, on budget, and beautifully executed - but that delivers absolutely nothing for the business. Eric Ries refers to this successful execution of a bad plan as “achieving failure”.
Ries also uses the term “success theatre” to describe charismatic individuals, often in larger organizations who are able to rally individuals and gather buy-in for projects that fail to deliver business value. Anyone can say “I have a vision for something big”, it’s far better to say “I have a vision for something big, and I’ve already run tests and proven that there is demand for it”.
There is surely nothing quite so useless as doing with great efficiency what should not be done at all.
Peter Drucker
When our world is changing and evolving, customers are changing, customer expectations are changing, marketing channels are changing and your company strategy is changing, a 100-year-old project management approach no longer works. Enter marketing experimentation.
The benefits of marketing experimentation
What is needed is a new approach to marketing project management, one that is appropriate for marketers and marketing teams today that operate under conditions of uncertainty.
As a result of work done on agile methodology in IT and software development over the years we know a few things to be true about the big bang, waterfall approach to projects:
-
Longer projects tend to get longer, they suffer from more scope creep and are more likely to get interrupted
-
Cost and time increase with complexity
-
There is often zero customer value delivered until right at the end
These problems are not unique to the IT world. The same applies when writing a book or essay, when writing code, and in many other areas of life. Every creative human endeavor requires an enormous amount of trial and error.
Big bang equals big risk, and if we can decrease the time between pivots and changing the direction, we can increase the odds of project success. We need to shift from resource optimization and instead focus on time-to-market optimization. Marketing experimentation approach enables this shift from fixed time versus fixed scope.
The more seldom we release something into the world, the more expensive and risky each release is. Conversely, the more often we release things, the cheaper and safer those releases become.
This is where marketing experimentation, and the minimum viable test comes in. For a deeper look at experimentation in SEO, see SEO Experiments: DIY, 3rd party service or in-house?. If you’re new to the concept of experimentation loops and want to understand how continuous testing and iteration can drive marketing success, check out our guide: What is an experimentation loop?. For a comprehensive introduction to building a culture of experimentation with AI, see AI experimentation strategies for marketing teams. To make testing the default rather than the exception, read how to build a culture of experimentation in your marketing team. And to benchmark how developed your programme is today, see the five stages of experimentation maturity.
Marketing experimentation & the minimum viable test
Humans will do marvellous things to avoid getting into the arena, where the possibility of failure is present. That’s why planning is so seductive. Planning can reduce uncertainty, and uncertainty is scary. But uncertainty can never be reduced to zero. So in most cases, it’s best to get momentum and solve the biggest problems as they come.
Unless you’ve released the same thing, to the same audience, at the same time before, you’re not really planning, you’re guessing. And if you’re going to be wrong, you’re better off spending $100 and losing a few days’ work, than spending $100,000 and losing 3 months of work.
This is why modern marketers run marketing experiments. Marketing experimentation is all about making specific, concrete predictions ahead of time in order to increase learnings and reduce uncertainty over the long term.
The goal of marketing experimentation is to move away from a big bet culture towards a more agile scientific approach to marketing with the ultimate goal of reducing waste.
Anyone can put compounds in a beaker and heat it up, in the same way that any marketing team can produce a piece of content - neither are science. The science comes from having a hypothesis, a set of predictions on what is likely to happen. The goal is for marketing to be effective, not purely efficient. Any marketing team can produce things in an efficient way, but only some are effective.
How to design and run a marketing experiment
A good marketing experiment follows the same six steps every time:
- Write the hypothesis. State the change, the result you expect and why, using a growth hypothesis.
- Pick one metric. Choose the number the test should move and record its baseline before you launch.
- Build the minimum viable test. Ship the smallest version that can prove or disprove the idea. See the minimum viable experiment.
- Run it for a fixed window. Set the end date up front and avoid interim verdicts.
- Read the result. Compare after with before, or against a control, and write down what you learned.
- Decide. Scale it, change it or stop, and log the outcome in a growth experiment template so the learning compounds.
Big-bang campaign vs marketing experiment
| Big-bang campaign | Marketing experiment | |
|---|---|---|
| Planning horizon | Months of upfront planning | Days, one small test |
| Cost of being wrong | High, discovered at launch | Low, capped at the size of the test |
| Learning speed | Once, after launch | Every test cycle |
| Evidence | Opinion and past playbooks | A hypothesis and a measured result |
| Decision rule | Stakeholder approval | A metric and threshold agreed in advance |
Marketing experiment examples
- Headline test: change one landing page headline and compare the sign-up rate. See A/B testing for when a split test is statistically valid.
- Landing page change: shorten a form and measure completions against last month.
- Channel test: spend a small fixed budget on a new channel for four weeks and track cost per lead.
- SEO title test: rewrite one page title around its top query and track position and clicks.
For more places to find ideas, see our guide to growth experiment ideas.
About Growth Method
Most teams that say they run marketing experiments are really running campaigns and hoping. Real experimentation needs a hypothesis, a recorded baseline and a result read against your own first-party data every time, and it has to be cheap enough to repeat weekly. That is the loop high-agency generalist marketers now run end to end: plan, launch, analyse, repeat.
Growth Method gives B2B marketing teams AI agents that run their campaigns: plan, launch and analyse, from live data, against one goal. Every campaign starts as a hypothesis, records its before-metrics at launch and is analysed against your goal, so the work that actually moves the needle compounds instead of getting lost in outputs.
Get started with Growth Method. First month free, no credit card required.
Frequently asked questions
What is marketing experimentation?
Marketing experimentation is testing a marketing idea on a small scale against a written hypothesis and a single metric, then using the result to decide whether to scale, change or stop it.
How is marketing experimentation different from A/B testing?
A/B testing is one tactic: a randomised split test. Marketing experimentation is the wider method, and it covers things you cannot split-test, such as branding, channels, content and SEO changes.
How do you design a marketing experiment?
Write a hypothesis, choose one metric and record its baseline, build the smallest test that can prove or disprove the idea, run it for a fixed window, then read the result and decide.
How long should a marketing experiment run?
Set the end date before you start. Many B2B tests run for two to four weeks, and low-traffic tests need longer. Avoid calling a result early.
How do you track marketing experiments?
Log the hypothesis, baseline, start and end dates, owner and result for every test in one place, tied to your team goal. A spreadsheet works to begin with. A platform such as Growth Method records baselines and results for you.