Home / Uncategorized / SEO Experiments: DIY, 3rd party service or in-house?

SEO Experiments: DIY, 3rd party service or in-house?

Article originally published in June 2019 by Stuart Brameld. Most recent update in April 2024.

Request a demo

Project management for growth and agile marketing professionals. Map your acquisition funnel, integrate analytics and run agile experiments.

Experiment results

Recent experiments results include competitor SEO, AI-driven content, exit-intent modals and AB testing homepage headlines.

Case study

"We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results." 


We are vetted mentors with Growth Mentor and a partner with the Agile Marketing Alliance.

Why Run SEO Experiments

Search Engine Optimisation (SEO) is the process of improving your website to ensure that when a potential customer searches for what you have to offer, you are one of the top results on their favourite search engine.

When it comes to SEO, companies tend to implement a set of SEO best practises, often supported by some kind of auditing tool such as Moz or Ahrefs.

Ahrefs Site Audit Tool

The problem with this approach is that you can spend months implementing these best practises (speeding up response times, getting new backlinks, adding structured data, updating alt tags and adding new content to pages) and it make absolutely no improvement to your search traffic.

“Traditionally, SEO tactics include trying out different known strategies and hoping for the best. You might have a good traffic day or a bad traffic day and not really know what triggered it, which often makes people think of SEO as magic rather than engineering.”

Julie Ahn, Pinterest Growth Engineer

If your organic search traffic in Google Analytics looks like the below, your SEO efforts aren’t working.

Google Analytics Report: No growth in search volumes for over 12 months

Just because something is a best practise doesn’t mean doing it will bring you more search traffic, and just because a change on somebody else’s website had a positive effect doesn’t mean it will have a positive effect for you. In short, cookie cutter “best practises” and big SEO audits don’t work and are not useful.

What is required is a data-driven approach that ensures, regardless of your team size, you are spending time and money on SEO activities that actually make a difference.

AB Testing

Most marketers are familiar with the concept of split testing (or a/b testing) where different versions of a webpage are tested to see which one performs better. This typically falls under the practise of Conversion Rate Optimisation or CRO.

To run one of these tests you split your website visitors into two groups using a tool such as Optimizely, VWO or Google Optimise, and show half of your visitors the original page whilst showing the other half the variant that the tool imposes.

  1. Control – the original page, with no changes
  2. Variant – the page with a change made e.g. an additional email form

If, over a period of time, the variant performs better based on some predetermined metric, such as more quote requests or form submissions, the variant becomes the new and improved regular site page.

The Challenge with SEO

When running paid search ads it is easy to a/b test content such as your ads and content in this way, and to attribute an increase or decrease in results to specific changes. Measurement is straightforward as there are relatively few variables, hence these tests provide a clear basis for decisions and further investment.

Unfortunately, measuring the effectiveness of Search Engine Optimisation changes is not so straightforward, hence why it is often done so poorly.

Duplicate Content

We can’t simply create two versions of one page and send half of Google’s traffic to one version and half to the other to see which one ranks better in the search engines, as this would result in duplicate (or near-duplicate) content.

Duplicate content is frowned upon by all search engines and, particularly when done at scale, can result in your site being demoted or removed from search results.

In the past some marketers have tried showing one set of content to humans, and a different set to search engine crawlers (such as Googlebot), to work around this. This is known as cloaking which is now firmly against the Google Webmaster Guidelines and will lead to penalties against your site in search rankings.

The Dynamic Search Environment

In addition to the duplicate content challenges above, Google is estimated to take into account over 200 different components and variables when determine search results, which obviously introduces a huge amount of variability to any test you may decide to run.


Many of these variables are completely outside of your control, in particular Google algorithm changes and the activity of your competitors, which hampers the ability to setup a controlled test.

As marketers we do not have control over many of the factors that determine how our website pages appear in search engine results. Variables in the daily search environment that affect SEO include:

  1. Lag times between when a page is crawled, and when it is processed
  2. Changes in search engine algorithms (on average two per day)
  3. Variations in search results based on location, time and login status
  4. The activity of competitor websites, such as significant ranking gains or drops
  5. A sudden change in backlinks for an individual page, for example, through a significant news event
  6. Changes to the internal linking structure, which can cascade through a site in unpredictable ways

Running Good SEO Experiments

Despite the above, with a good methodology it is possible to decrease the effects of these variables in order to run high quality, valid SEO experiments. Companies such as Pinterest, Etsy and Thumbtack all perform regular SEO experiments that have led to huge increases in search traffic.

There are hundreds of different ways to do SEO, including sitemaps, link-building, search-engine-friendly site design and so on. The best strategy for successful SEO can differ by product, by page and even by season. Identifying what works best for each case helps us move fast with limited resources. By running a large number of experiments, we found some well-known strategies for SEO didn’t work for us, while certain tactics we weren’t confident about worked like a charm.

Julie Ahn, Pinterest Engineer, Growth Team

The best introduction to running SEO experiments is this short video from Rand Fishkin, founder and former CEO of Moz.

How Do I Successfully Run SEO Tests on My Website(s)?

SEO Split Testing

Whilst a/b testing for Conversion Rate Optimisation is user oriented with the goal of getting more people to convert whilst on your page, a/b testing for SEO is page oriented with the goal of driving more traffic to your page in the first place.

To run SEO experiments, instead of creating groups of users we create groups of pages. This is both:

  1. Safe, as from a search engine perspective as there is just one version of each page, and that same page is shown to regular users and GoogleBot alike and
  2. Effective, because it isolates the change being made.
Image showing how we bucket users for traditional ab tests, and pages or SEO tests
Groups not Users

SEO split tests allow us to test whether if we make a change to one group of pages, whether more people will click the links in the search results and visit your site.

For example, you might change the title tags on 50% of your product pages, or blog posts, and see how they perform compared to the other half.

On overview of the SEO split testing process is outlined below:

  1. Pick an SEO element for testing
  2. Identify the set of pages you want to improve
  3. Randomly group the pages into control and variant groups
  4. Make the SEO-focused change on the pages in the variant group
  5. Measure the resulting changes in organic search traffic and click-through rate

Implementing SEO Split Tests

SEO split testing is typically implemented in 1 of 3 ways, listed below:

  1. Do It Yourself
  2. Using a 3rd party service
  3. Building an in-house SEO experimentation platform

1. The DIY Approach

To split-test without the help of testing software, you can manually group the pages yourself and monitor traffic in Google Search Console. This is typically how most marketing teams start out with SEO experimentation as it’s a low cost way to get started.

The process here is as follows:

  1. Divide site pages into two groups: control and variant.
  2. Make a change to the variant group.
  3. Wait 21–28 days while monitoring clicks, impressions, and CTR with Search Console.
  4. Collect results for control pages
  5. Do the same for your variant group pages.
  6. Compare the variant and control metrics to determine the largest improvement. The pages with the highest CTR improvements are the winners, and you should be able to determine whether the SEO change was beneficial or not

Unfortunately most content management systems (CMS’s) do not offer the ability to make changes to arbitrary groups of pages hence, depending on what you are testing, it will require some manual effort.

2. Use A 3rd Party Service

In recent years a number of third-party tools have come to market that provide extensive SEO testing capabilities, the best known of which are:

  1. Distilled ODN https://odn.distilled.net/
  2. Rank Science https://www.rankscience.com/
  3. AB Rankings https://abrankings.com/
  4. Rank Search https://www.ranksense.com/

These services typically sit in front of your web server acting in a similar way to how a content delivery network (CDN) operates. They provide the ability to make arbitrary changes to the HTML of any page or group of pages on your website, create an experiment and calculate statistically significant results to continually test what provides the best results in terms of organic traffic.

3. Build An In-House Platform

Companies with significant organic web traffic may choose to build their own in-house SEO experimentation platform, which is exactly what the Pinterest engineering team have done.

This obviously requires significant in-house expertise and resource, and is likely only a viable option for larger web teams. The Pinterest in-house platform provides three key components:

  1. The ability to define experiments and group pages
  2. The ability to compute traffic to the pages in each experiment group on a daily basis
  3. A Dashboard to view results

SEO Experiment Best Practises

Whichever method you choose, the following best practises will ensure your SEO a/b tests deliver accurate results:

  1. Avoid keywords and terms that have significant volatility in the search engine rankings. Aim for less active search terms where there is less movement, churn, and competition.
  2. In order to ensure you can clearly analyse the difference when a new variant is launched, avoid tests with pages where traffic levels fluctuate significantly.
  3. When the experiment starts, make an annotation in Google Analytics
  4. Keep things small (and reversible) to reduce the likelihood of any long term traffic issues or penalties. Avoid tests on your highest ranking pages.
  5. Make your changes to similar groups of pages (for example all product pages) so that the control group and the test group are as similar as possible.
  6. Choose terms where you currently rank on pages 2 or 3 in the search results. Increasing your ranking on page 1 is hard, and beyond the third page of results, small changes can lead to movements of 10 or more places.
  7. Tests should, ideally be repeatable, and repeated, in order to provide assurance that what was changed was the impacting factor. If you stop the tests, do search results fall backdown? If you turn on again, do they go back up?
  8. Be sure to regularly update your sitemap and prune unnecessary pages, this increases the overall quality of content on your site, and preserves crawl budget on larger sites.

Good SEO Experiments

In general it is best practise to a/b test large-scale items across all pages, and to test changes to pages that already rank in search results with a steady amount of traffic.

Pinterest, arguably the leaders in SEO experimentation, suggest considering running experiments in the following areas:

Title Tag KeywordsYour title tag is the headline for your search result. Test narrative versus keyword-driven titles & different formats.
Title Tag LengthDo shorter title tags perform better than long ones? See this article from the Thumbtack engineering team.
Meta description LengthDo longer or shorter meta descriptions perform better?

Longer descriptions take up more real estate in search results and can give the appearance of more authority.
Scale the number of pagesIs there potentially to automatically scale the number of pages on your site.

Zapier have specific pages for each integration totalling over 500,000 pages.
Content formatTest your core page layouts and templates. Which layouts perform best for SEO?
Amount of content Does adding more content (text and images) help pages rank higher in search?

What if you add an additional heading to your page content?
Test Language Specific PagesBuild translations of language specific landing pages to rank for non-English search queries.

Additional experiments you may like to consider include:

  • Internal Link Blocks – test adding or removing blocks of links, such as WordPress related post modules
  • Footer Links – test adding or removing links from your website footer
  • Social Buttons – test adding/removing social, commenting and share buttons
  • Structured Markup – what is the impact of adding or removing structured data?

Poor SEO Experiments

Running tests for specific pages or search terms, particularly for high competition keywords, is harder to test and as such should be avoided.

Additionally avoid link based experiments as they are almost impossible to prove. In an ideal world a successful link experiment would be one where you add links, and see your search results go up, then remove the links, and see them fall back down. However, Google tends to consider the value of links (positive or negative) even after they have been removed. This is known as ‘The Ghost Effect’, and makes running accurate link-based experiments almost impossible.

Measuring Results

SEO Experiments typically focus on the following metrics, with Organic Traffic (Sessions) and SERP Rankings being the 2 most important:

Organic TrafficSearch traffic (organic sessions) is an excellent success metric for SEO experiments.
SERP RankingRanking/position (the average for all pages in your groups) can be used alongside organic traffic as changes here can often be detected more quickly.
Total ImpressionsHow many people are seeing your page on the SERP results.
Click-Through Rate (CTR)How many people clicked on your search result? You may also wish to measure total clicks.

Remember when checking on SERP Rankings to always check logged out, non-personalised and non geo-biased results.

Google Search Console

For a well-run test the two main metrics you should track are:

  1. Total organic entrances (or sessions) day by day for the sum of your control set of pages.
  2. Total organic entrances (or sessions) day by day for the sum of your variant set of pages.

Google Search Console provides the ability to filter for a group of pages and compare metrics across date ranges, such as 4 weeks before the change, and 4 weeks after.

Advanced Considerations

Traffic Volume & Page Grouping

Ensuring your groups of pages are as balanced as possible for testing is challenging because visits to individual pages are often highly volatile with many fluctuations on a daily basis. A sudden spike in search volume for an individual page may be the result of increased demand for the page (a political event, or January sales) as opposed to the page performing better.

As a result, SEO split testing is better suited to larger sites with many of the same types of pages, in order to control for these individual page variations within each group.

If you have a small website with few of the same type of page and/or low search traffic, you can still get started with SEO experimentation by making safe changes to your site without split-testing them:

  1. Make a change to one or a few pages that get reasonable traffic.
  2. Monitor immediate changes to detect dips in traffic, then wait a defined period of time (at least 28 days).
  3. Use Search Console to assess changes in impressions, clicks, and CTR before and after the changes.
  4. Iterate as needed and roll out successful changes to more pages.

SEO Experiment Timings

One advantage of SEO testing is that search engines are more rational and consistent than the collection of human visitors that decide the outcome of an on-page a/b test. This means that, assuming there are no significant algorithm updates that effect your experiment, you should quickly be able to ascertain whether anything dramatic is happening as a result of a test.

Often the impact of an experiment on traffic starts to show as early as a couple of days after launch. However, it is recommended to run your test and gather data for 4 to 6 weeks in order to allow for search engine results to stabilise. You want to allow plenty of time to ensure Google has picked up on all the changes on your page and has re-indexed the impacted pages.

Useful Links

We have provided some additional links below should you wish to read more into this topic: