Table of contents
Why Run SEO Experiments
Search Engine Optimisation (SEO) is the process of improving your website to ensure that when a potential customer searches for what you have to offer, you are one of the top results on their favourite search engine.
When it comes to SEO, companies tend to implement a set of SEO best practises, often supported by some kind of auditing tool such as Moz or Ahrefs.
The problem with this approach is that you can spend months implementing these best practises (speeding up response times, getting new backlinks, adding structured data, updating alt tags and adding new content to pages) and it make absolutely no improvement to your search traffic.
Traditionally, SEO tactics include trying out different known strategies and hoping for the best. You might have a good traffic day or a bad traffic day and not really know what triggered it, which often makes people think of SEO as magic rather than engineering.
Julie Ahn, Pinterest Growth Engineer
If your organic search traffic in Google Analytics looks like the below, your SEO efforts aren’t working.
Just because something is a best practise doesn’t mean doing it will bring you more search traffic, and just because a change on somebody else’s website had a positive effect doesn’t mean it will have a positive effect for you. In short, cookie cutter “best practises” and big SEO audits don’t work and are not useful. For a deeper dive into how and when SEO works, including key considerations for your business, see Search Engine Optimisation: The Good, The Bad & The Ugly.
What is required is a data-driven approach that ensures, regardless of your team size, you are spending time and money on SEO activities that actually make a difference. One useful tool to support this is the new Growth Method & Plausible integration, which helps connect SEO results to experiments and activities. Another effective strategy is Pain Point SEO, which focuses on lower-volume, high-intent keywords that closely align with user challenges and buying signals.
AB Testing
Most marketers are familiar with the concept of split testing (or a/b testing) where different versions of a webpage are tested to see which one performs better. This typically falls under the practise of Conversion Rate Optimisation or CRO.
To run one of these tests you split your website visitors into two groups using a tool such as Optimizely or VWO, and show half of your visitors the original page whilst showing the other half the variant that the tool imposes. Google Optimise, once a popular free option here, was shut down in 2023; teams looking for a lower-cost alternative now tend to use open-source tools like GrowthBook instead.
-
Control - the original page, with no changes
-
Variant - the page with a change made e.g. an additional email form
If, over a period of time, the variant performs better based on some predetermined metric, such as more quote requests or form submissions, the variant becomes the new and improved regular site page.
The Challenge with SEO
When running paid search ads it is easy to a/b test content such as your ads and content in this way, and to attribute an increase or decrease in results to specific changes. Measurement is straightforward as there are relatively few variables, hence these tests provide a clear basis for decisions and further investment.
Unfortunately, measuring the effectiveness of Search Engine Optimisation changes is not so straightforward, hence why it is often done so poorly.
Duplicate Content
We can’t simply create two versions of one page and send half of Google’s traffic to one version and half to the other to see which one ranks better in the search engines, as this would result in duplicate (or near-duplicate) content.
Duplicate content is frowned upon by all search engines and, particularly when done at scale, can result in your site being demoted or removed from search results.
In the past some marketers have tried showing one set of content to humans, and a different set to search engine crawlers (such as Googlebot), to work around this. This is known as cloaking which is now firmly against the Google Webmaster Guidelines and will lead to penalties against your site in search rankings.
The Dynamic Search Environment
In addition to the duplicate content challenges above, Google is estimated to take into account over 200 different components and variables when determining search results, which obviously introduces a huge amount of variability to any test you may decide to run.
Many of these variables are completely outside of your control, in particular Google algorithm changes and the activity of your competitors, which hampers the ability to setup a controlled test.
As marketers we do not have control over many of the factors that determine how our website pages appear in search engine results. Variables in the daily search environment that affect SEO include:
-
Lag times - between when a page is crawled, and when it is processed
-
Search engine algorithm changes - on average two per day
-
Search result variations - based on location, time and login status
-
Competitor activity - such as significant ranking gains or drops
-
Backlink changes - a sudden change in backlinks for an individual page, for example, through a significant news event
-
Internal linking changes - which can cascade through a site in unpredictable ways
Running Good SEO Experiments
Despite the above, with a good methodology it is possible to decrease the effects of these variables in order to run high quality, valid SEO experiments. Companies such as Pinterest, Etsy and Thumbtack all perform regular SEO experiments that have led to huge increases in search traffic.
There are hundreds of different ways to do SEO, including sitemaps, link-building, search-engine-friendly site design and so on. The best strategy for successful SEO can differ by product, by page and even by season. Identifying what works best for each case helps us move fast with limited resources. By running a large number of experiments, we found some well-known strategies for SEO didn’t work for us, while certain tactics we weren’t confident about worked like a charm.
The best introduction to running SEO experiments is this short video from Rand Fishkin, founder and former CEO of Moz.
SEO Experiments in the AEO Era
The rise of AI Overviews and answer engines like ChatGPT, Perplexity and Gemini hasn’t replaced the need for rigorous SEO experiments, it’s added a second measurement problem on top of the first. Alongside organic clicks and rankings, you now need to track whether your content is being retrieved, cited and mentioned by AI systems, since a page can lose click-through even while gaining AI citations.
The same experimental discipline described in this article for classic SEO applies to AEO: define a hypothesis, isolate a single change, measure before and after, and be honest about confounds like algorithm updates. What changes is the metric set. Alongside organic sessions and average position, track citation share and AI referral traffic. See our full AEO guide for the tactics and measurement framework in detail.
Which Approach Is Right for You?
Whether you run SEO experiments yourself, hire a specialist service, or build the capability in-house depends on your team’s speed, budget and how much control you want over the testing process.
| Approach | Cost | Speed | Control | Expertise required | Best for |
|---|---|---|---|---|---|
| DIY | Low (your team’s time) | Slow to start, faster once the framework exists | Full control over hypotheses and rollout | Needs someone comfortable with experiment design and stats | Teams with an existing growth culture and a technical marketer or analyst on staff |
| 3rd-party service | Medium to high (agency/consultant fees) | Fast to start, dependent on the vendor’s queue | Shared control; you brief, they execute | Minimal in-house expertise needed | Teams that want SEO experimentation now but lack the internal skill set |
| In-house build | High upfront (tooling, headcount) | Slow to start, fastest to iterate once built | Full control end-to-end | Needs a growth engineer or similar | Larger teams running enough experiments to justify dedicated tooling and a person to own it |
There is no universally right answer here: a five-person team validating its first few SEO hypotheses is usually better served going DIY or hiring a service for a defined engagement, while a team running dozens of experiments a year benefits from the control and speed an in-house build eventually provides.
About Growth Method
Growth Method is the agentic marketing platform for B2B teams: plan your strategy, ship campaigns, and learn what works, all in one place, for people and agents. Running a proper SEO experiment, exactly the kind of test-and-learn work this article describes, is precisely what our campaigns feature is built for. Hypothesise a change to a page, launch it with the baseline already recorded, then let an agent read the before/after against organic sessions, CTR and average position instead of you guessing whether it worked.
We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results.
Laura Perrott, Colt Technology Services
Get started to turn your next SEO experiment into a tracked campaign, not a guess.
Frequently asked questions
What’s the difference between an SEO experiment and general SEO best practice?
Best practices are things you do because they’re broadly known to help, such as fast pages, clean structured data, or fresh content. An SEO experiment tests whether a specific change actually moves your traffic, since best practices don’t work uniformly for every site or page.
Can I A/B test SEO the same way I A/B test a landing page?
Not directly. You can’t show two versions of the same URL to Google and split traffic between them without risking duplicate content penalties. Instead, SEO experiments typically split by page, where a group of similar pages gets the change and a comparable group doesn’t, or run as a time-based before/after test on a single page.
Should I use a DIY approach, a 3rd-party service, or build in-house tooling for SEO experiments?
It depends on team size and volume. A DIY approach or a defined engagement with a specialist service suits teams running occasional experiments. An in-house build, with dedicated tooling and a growth engineer, only pays off once you’re running enough tests to justify it.
Does Answer Engine Optimisation (AEO) replace the need for SEO experiments?
No. AEO adds a second thing to measure, namely citations and AI referral traffic, but the same experimental discipline of a hypothesis, an isolated change, and before/after measurement still applies.
What happened to Google Optimise?
Google shut down Google Optimise in September 2023. Teams that used it for SEO or CRO experiments have largely moved to Optimizely, VWO, or open-source alternatives like GrowthBook.
How long should an SEO experiment run before I read the results?
Long enough to rule out normal variance and let Google fully re-crawl and re-rank the affected pages, typically a minimum of 8 to 12 weeks, longer if a core algorithm update falls inside your test window.