Table of contents
Most marketing campaigns barely move the number they were aimed at. A few move it a lot. That uneven pattern is what people usually mean by the CRO power law, and it should change how you choose what to run next.
The term has two meanings that are often mixed up. Below is the evidence for each, and what it means for how you prioritise.
What is the CRO power law?
A power law describes a relationship where a small number of inputs account for most of the output. The 80/20 rule (the Pareto principle) is the best-known example. In conversion rate optimisation (CRO) the phrase is used in two ways:
- The Power Law of CRO. A term coined by the agency Conversion Rate Experts. It says profit rises disproportionately as your conversion rate improves, because several benefits multiply together.
- The power law of results. The observation, backed by large-scale experiment data, that a small share of tests and campaigns produce most of the total gains.
The first explains why improving conversion is worth the effort. The second explains why most of your attempts will not be the ones that pay off. You need both to prioritise well.
The Power Law of CRO: why conversion gains compound
Conversion Rate Experts argue that a better conversion rate helps you in three ways at once:
- More customers from the same traffic. Revenue is visitors multiplied by conversion rate multiplied by lifetime customer spend. Double the conversion rate and revenue doubles, with no extra ad spend.
- Profit grows faster than revenue. When conversion doubles, your variable costs double, but your ad costs and fixed costs stay the same. In their illustration, doubling the conversion rate multiplies profit by four.
- You can afford more traffic. A higher profit per visitor lets you bid more for ads, pay affiliates more, and enter channels that were too expensive before.
Because the three benefits multiply rather than add, they conclude that “your profit has a power-law relationship to your conversion rate.”
Image source: Conversion Rate Experts
Treat this as a model, not a measurement. Conversion Rate Experts is an agency that sells CRO, the example is illustrative arithmetic, and they note that the multipliers depend on the financials of the business. Their companion study applies pay-per-click data from one real business to a fictional company. The logic is sound, but run the numbers with your own margins before quoting a multiplier.
The power law of results: a few wins drive most of the gains
The second meaning is about outcomes. If you run 50 campaigns in a year, the gains will not be spread evenly across them. A few will account for most of the improvement and the rest will be flat or negative.
The best public evidence comes from Microsoft Bing. In A/B Testing with Fat Tails, a team of economists and Microsoft researchers (Eduardo Azevedo, Alex Deng, José Luis Montiel Olea, Justin Rao and Glen Weyl) analysed experiments run on Bing’s experimentation platform and measured how concentrated the gains were:
“In particular, the top 2% of ideas are responsible for 74.8% of the historical gains. This is an extreme version of the usual 80-20 Pareto rule.”
Eduardo Azevedo and co-authors, A/B Testing with Fat Tails, August 2019
The quote is from the authors’ August 2019 version of the paper, which was later published in the Journal of Political Economy. Two caveats apply. Bing is a search engine with enormous traffic, and these were product changes, not marketing campaigns. Your split will not be 2% and 74.8%. What transfers is the shape: results concentrate in a few outliers.
Most ideas fail, and experts can’t pick the winners
Ron Kohavi, who led experimentation at Microsoft and Amazon, and Harvard Business School professor Stefan Thomke start from John Wanamaker’s line about half his advertising being wasted, then add:
“We’ve found something similar to be true of new ideas: The vast majority of them fail in experiments, and even experts often misjudge which ones will pay off. At Google and Bing, only about 10% to 20% of experiments generate positive results.”
Ron Kohavi and Stefan Thomke, The Surprising Power of Online Experiments, Harvard Business Review, 2017
At Microsoft as a whole, they report, about one third of experiments prove effective, one third are neutral and one third have negative results. Their conclusion is that companies “need to kiss a lot of frogs (that is, perform a massive number of experiments) to find a prince.”
The biggest win looked like a minor idea
The same article opens at Bing. In 2012 an employee suggested changing how ad headlines were displayed. It was one of hundreds of ideas, program managers rated it a low priority, and it sat untouched for more than six months. An engineer eventually ran a simple A/B test because the code was cheap to write.
Revenue rose 12%, worth more than $100 million a year in the United States alone. In the authors’ words: “It was the best revenue-generating idea in Bing’s history, but until the test its value was underappreciated.”
A manual prioritisation process had ranked the best idea the team ever had near the bottom of the list.
Small wins still count
The power law does not mean small wins are worthless. Kohavi and Thomke also write that “most progress is achieved by implementing hundreds or thousands of minor improvements.” Both things are true at once: many small gains add up, and a rare outlier changes the year. Neither can be reliably predicted in advance, which is the argument for volume.
The power law applies to channels too
Results concentrate by channel as well as by campaign. Peter Thiel makes the point in the distribution chapter of Zero to One:
“Most businesses get zero distribution channels to work: poor sales rather than bad product is the most common cause of failure. If you can get just one distribution channel to work, you have a great business. If you try for several but don’t nail one, you’re finished.”
Peter Thiel, co-founder of PayPal and Palantir, Zero to One, 2014
This fits with running more campaigns. You run many small campaigns to find the channel that works, then concentrate your effort there instead of spreading it evenly.
What the power law means for how you prioritise
- Run more campaigns. If a small share of campaigns produce most of the gains and you can’t tell which ones in advance, the number you run is the lever you control. The Bing paper reaches the same conclusion: when results are very fat tailed, a lean strategy of trying more ideas beats a few big bets.
- Keep each campaign small. The cheaper a campaign is to run, the more of them you can run. Start with a minimum viable test and scale up the ones that show promise.
- Size ideas against your goal. An idea whose best case would barely move your metric can’t be the outlier, however easy it is. Ask what the upside is if it works, and compare that with the gap you need to close.
- Be sceptical of impact and confidence scores. Frameworks such as ICE and RICE ask you to guess impact and confidence up front, which is the judgement the evidence says even experts get wrong. See our comparison of prioritisation frameworks for where each one helps and where it misleads.
- Double down on winners. When a campaign lands, treat it as a signal. Extend it and move budget toward it before going back to the backlog.
- Record every result. Flat and negative results tell you where the outliers are not, which is only useful if the next person can find them.
About Growth Method
The power law is the reason prioritisation matters, and the reason gut-feel scoring is not good enough.
Growth Method is the agentic marketing platform for B2B teams. Agents plan, launch and analyse campaigns from your live marketing data, so you can run more campaigns. Every new idea is scored automatically on how easy it is to run and how relevant it is to your goal. When your goal has a connected metric, scoring also sees the gap still to close, so ideas too small to matter drop down the backlog.
We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results.
Laura Perrott, Colt Technology Services
Get started to run more campaigns and find the few that drive your growth.
Frequently asked questions
What is the CRO power law?
The term is used in two ways. Conversion Rate Experts coined the Power Law of CRO to describe how profit rises disproportionately as conversion rate improves, because more customers, better margins and cheaper traffic multiply together. The phrase is also used for the power law of results: a small share of tests and campaigns produce most of the total gains.
Does the 80/20 rule apply to A/B tests and marketing campaigns?
The pattern does, and the real split can be more extreme. A study of experiments on Microsoft Bing found the top 2% of ideas were responsible for 74.8% of the historical gains. Your numbers will differ, because Bing is a search engine with enormous traffic, but results concentrated in a few outliers is the norm rather than the exception.
How many marketing campaigns should you expect to win?
Fewer than most teams plan for. Ron Kohavi and Stefan Thomke report that only about 10% to 20% of experiments at Google and Bing generate positive results, and that at Microsoft as a whole roughly one third help, one third are neutral and one third hurt. Plan for most campaigns to be flat and judge the programme by its few big wins.
How do you prioritise when you can’t predict the winners?
Run more campaigns, keep each one small, and rank ideas on things you can judge: how easy the idea is to run and whether its best case is big enough to move your goal. Be sceptical of gut-feel impact and confidence scores, because the evidence shows even experts misjudge which ideas pay off.