Skip to content

A/B test calculators: how they compare to an LLM and your testing tool

Stuart Brameld

Stuart Brameld

Founder
Contents

Every A/B test ends with the same question: did the change work, or is this noise?

There are now three common ways to answer it. You can paste the numbers into a free A/B test calculator. You can trust the stats engine inside your testing tool. Or, increasingly, you can drop the results into ChatGPT or Claude and ask.

Each one has a place. This guide compares five of the calculators CRO teams use, including one built for AI agents, then sets out the pros and cons of a calculator against an LLM and against your testing tool, so you know which to reach for and when.

Table of contents

What an A/B test calculator does

A good A/B test calculator does two jobs.

Before the test (pre-test analysis). You give it your baseline conversion rate, traffic and the smallest lift worth detecting, and it tells you the sample size and how long the test needs to run. This is where most tests are won or lost: a test without enough traffic can’t find a realistic effect, however good the idea. Our guide to the reverse MDE calculator covers this step in depth.

After the test (test evaluation). You give it visitors and conversions for each variant, and it tells you whether the difference is statistically significant, with a p-value, a confidence interval or a probability that the variant beats control.

The best calculators also catch the problems that quietly invalidate a result: a broken traffic split (sample ratio mismatch), too many variants compared at once, or a metric like revenue per visitor that needs different maths from a simple conversion rate.

If you’re new to the underlying concepts, start with what is A/B testing and our longer A/B testing guide.

Five A/B test calculators compared

CalculatorPriceBest forPre-testPost-testStand-out feature
ABTestGuideFreeQuick significance checksYesYes (frequentist), plus a separate Bayesian toolVisual distributions, shareable result links
Analytics ToolkitPaid, 30-day free trialRigorous programmes and revenue metricsYes, ROI-based planningYes, including non-binomial metricsAGILE sequential testing
SpeeroFreePlanning test capacity and showing business impactYes, MDE by weekYes, Bayesian or Z testSRM alert and revenue forecast
OptimizelyFreePlanning tests you’ll run in OptimizelyYesNoMatches Optimizely’s sequential Stats Engine
ABTestResultFree web calculators; MCP server free for personal useLetting an AI agent run the statisticsYesYes, frequentist or BayesianMCP server that runs locally

ABTestGuide (Online Dialogue)

ABTestGuide comes from the Dutch experimentation community around Online Dialogue and Conversion Hotel, and it has been a CRO bookmark for years. It does one thing very well: tell you quickly whether a finished test is significant, and show you why.

You enter visitors and conversions for A and B, choose a one-sided or two-sided hypothesis and a confidence level of 90%, 95% or 99%, and it returns the relative uplift, p-value, Z-score and standard errors, with a chart of the two overlapping distributions. A pre-test mode estimates what your traffic can detect, and sister tools cover test size, Bayesian analysis and false discovery rate.

Pros

  • Free, fast and needs no account
  • The distribution chart makes significance easy to explain to a stakeholder
  • Shareable result URLs, so the evidence travels with the decision
  • Separate Bayesian calculator if your team prefers “probability to beat control”

Cons

  • Conversion rates only: no revenue per visitor or average order value
  • Two variants at a time, with no correction for A/B/n tests
  • No sample ratio mismatch check
  • It reports “observed power” after the test, which is easy to misread as proof the test was big enough

Analytics Toolkit (Georgi Georgiev)

Analytics Toolkit is built by Georgi Georgiev, author of Statistical Methods in Online A/B Testing, and it is the most statistically rigorous option here. It’s less a single calculator than a set of them: a test planner, statistical significance, sample size, power and MDE, multiple comparisons, meta-analysis and a sample ratio mismatch checker.

Its headline feature is AGILE, a sequential testing method that lets you check results at planned points and stop early for a clear winner or loser without breaking the statistics. The test planner also sizes tests against business returns rather than an arbitrary MDE.

Pros

  • Handles non-binomial metrics such as revenue per user and average order value, from an uploaded file
  • Corrects A/B/n tests automatically (Dunnett’s correction) and supports non-inferiority tests
  • Built-in SRM check, plus meta-analysis to combine results across tests
  • Sequential testing means faster decisions without peeking problems
  • Excellent documentation and a deep blog if you want to understand the maths

Cons

  • Paid after a 30-day free trial, which is hard to justify for a team running a few tests a quarter
  • More settings and concepts to learn than a free calculator
  • Built for analysts, so less friendly for a marketer who just wants a yes or no

“Statistical significance measures how likely it would be to observe what was observed, assuming the null hypothesis is true.”

Georgi Georgiev, founder of Analytics Toolkit

Speero

Speero’s A/B test calculator (Speero is the experimentation agency formerly known as CXL Agency) puts pre-test and post-test analysis in one place. That matters more than it sounds: Speero points out that mixing a sample size calculator from one site with a duration calculator from another gives conflicting answers, because they use different underlying statistics.

The pre-test tab shows the MDE you can detect if you run for one to six weeks, which is the most useful single view for deciding whether a page has enough traffic to test. The post-test tab lets you choose Bayesian or Z test (see frequentist vs Bayesian for which to pick), one-sided or two-sided, warns you about a possible sample ratio mismatch, and turns the lift into extra monthly transactions and a six-month revenue forecast. There’s also a free Chrome extension that pulls traffic and conversions straight from GA4.

Pros

  • Free, with planning and analysis in one consistent tool
  • MDE by week answers “can we even test this?” in seconds
  • SRM alert catches broken traffic splits
  • Revenue impact makes results easier to sell internally
  • GA4 Chrome extension saves copying numbers across tabs

Cons

  • Binomial metrics only for now; Speero says non-binomial support is still on its list
  • The MDE view stops at six weeks, which is short for low-traffic B2B pages
  • No sequential testing, so the result is only valid once the planned sample is reached

Optimizely

Optimizely’s sample size calculator is a planning tool, not a significance checker. You enter a baseline conversion rate, a minimum detectable effect and a significance level, and it returns the sample size per variation.

The difference is under the hood. It uses the same formula as Optimizely’s Stats Engine, a sequential test with false discovery rate controls. Optimizely says that if the true effect is larger than your MDE, a test may finish up to twice as fast as a fixed-sample test, and that the sample size is a planning estimate rather than a stopping rule.

Pros

  • Free and very simple: three inputs, one answer
  • If you run tests in Optimizely, the estimate matches how your results will be calculated
  • Good plain-English FAQs on MDE and baseline rates

Cons

  • No post-test analysis, so it can’t tell you whether a finished test won
  • No statistical power setting, because the Stats Engine handles power differently
  • Numbers won’t match a classic fixed-sample calculator, which confuses teams that use both
  • Most useful to Optimizely customers; less so for everyone else

ABTestResult

ABTestResult is a set of free web calculators (significance, sample size, Bayesian, duration, ROI and MDE) built by Andrea Corvi, Head of Experimentation at Livescore Group. What sets it apart is its MCP server, which gives an AI assistant such as Claude, ChatGPT or Cursor the same calculators as tools it can call.

The server covers the whole test lifecycle: sample size and MDE for design, an SRM check for health, a Z-test for conversion rates and a Welch’s T-test for averages such as revenue per visitor, paired before and after tests, and a Bayesian analysis with probability to be best and expected loss. It runs on your own machine with one line of config, no account and no API key, and each result comes back with a link that opens the matching web calculator with your inputs filled in, so a person can check the agent’s work.

Pros

  • The AI calls real statistics code instead of generating numbers, so the same inputs give the same answer every time
  • Covers revenue and other averages as well as conversion rates
  • SRM check and both frequentist and Bayesian analysis in one place
  • Test data stays on your machine; only anonymous usage telemetry (tool name and client type) is sent
  • Every result links back to a web calculator for a human check

Cons

  • The MCP server is source-available, not open source: its licence covers personal, non-commercial use, and business or client use needs a separate licence from ABTestResult
  • No sequential testing or multiple-comparison correction, so the agent still has to wait for the planned sample and avoid comparing many variants at once
  • It only answers the questions the agent asks; an agent that never calls the SRM check never sees the broken split

A/B test calculator vs throwing your results into an LLM

Pasting results into ChatGPT, Claude or Gemini is now the default for a lot of marketers. It’s quick, it explains itself in plain English, and it will happily tell you whether to roll out the winner. The question is whether you can trust it.

The honest answer: modern models usually calculate the p-value correctly, especially when they run the maths in code rather than in their head. A recheck of Claude’s answers against statsmodels found the Z-score and p-value matched to four decimal places. But the same test found the required sample size drifted between “roughly 350 to 400” and “400 to 450” across runs, with no working shown. The maths is rarely the problem. The judgement around it is.

Pros of using an LLM

  • Plain-language explanation. It can translate a confidence interval into a sentence your CMO will understand
  • Business context. Ask it what a 0.4 point lift means in extra leads or revenue per month, and it will do the arithmetic
  • Flexible inputs. It can work from a messy export, a screenshot or several segments at once
  • Good at follow-up questions. “What would make this result untrustworthy?” often surfaces risks you hadn’t considered

Cons of using an LLM

  • Inconsistent answers. Ask the same question twice and sample size estimates, intervals and wording can change
  • It answers the question you asked. It rarely volunteers that your test was underpowered, that you stopped early or that your traffic split looks broken
  • Silent method choices. It may pick a one-sided test, skip a multiple-comparison correction on an A/B/n test or use sessions where you meant users, without saying so
  • It thinks in thresholds. Research from Kellogg found ChatGPT, Gemini and Claude treat p = 0.049 and p = 0.051 as opposite outcomes, just as people do
  • No audit trail. A chat transcript isn’t a reproducible calculation you can share and recheck

“As with people, this ‘dichotomania’ seems deeply embedded in the way the AI models respond.”

Blake McShane, Professor of Marketing, Kellogg School of Management

How to use an LLM safely

Use the LLM for interpretation, and a calculator for the verdict. In practice:

  1. Give it raw counts, not percentages: visitors and conversions per variant, the unit (users or sessions) and how long the test ran
  2. Tell it your plan: the MDE and sample size you set before launch, and whether anyone stopped the test early
  3. Ask it to run the calculation in code and show every step, rather than estimate
  4. Ask it to check SRM, power against your MDE and multiple comparisons before giving a verdict
  5. Check the verdict in a calculator such as ABTestGuide or Speero before you act on it

Better still: let the AI call a calculator

The steps above keep one person honest. They break down once AI agents propose, build and launch tests for you, because the inconsistency repeats on every test. One run sizes the test properly, the next stops it early. Across dozens of tests a month, those slips add up to winners that were never real.

The fix is to take the maths out of the model. Give the agent a calculator as a tool through an MCP server, such as ABTestResult’s, and let the LLM do what it is good at: deciding which question to ask and explaining the answer.

“The fix isn’t a smarter model. It’s taking the stats out of the model’s head and making it a tool every agent calls the same way.”

Andrea Corvi, Head of Experimentation, Livescore Group, and creator of ABTestResult

Tell the agent which checks to run every time (SRM, power against your MDE, then significance), so the tool is called the same way on every test.

A/B test calculator vs using your testing tool

Every testing platform, from VWO and Convert to Optimizely and PostHog, comes with its own stats engine. So why use a separate calculator at all?

Pros of using your testing tool’s stats

  • Built for the data it collects. The engine knows exactly how visitors were bucketed, which removes a whole class of copy-paste errors
  • Often more advanced. Many tools now use sequential testing or Bayesian methods that make early looks safe, which a basic calculator doesn’t
  • No extra step. Results update live in the dashboard your team already watches
  • Guardrails built in. Better tools flag SRM, novelty effects and multiple goals automatically

Cons of using your testing tool’s stats

  • Every vendor does it differently. Methods, defaults and terminology vary, so a “95% winner” in one tool isn’t comparable with another. If you switch tools or run tests in several, your historical results stop lining up
  • It only sees its own metrics. Pipeline, qualified leads and revenue usually live in your CRM or analytics, not the testing tool, and those are the growth goals that matter
  • The dashboard invites peeking. A live significance figure tempts people to stop the moment it turns green. Unless the tool uses sequential statistics, that inflates false positives
  • Black-box incentives. A vendor has a reason to show you winners. An independent calculator is a cheap second opinion on a surprising result
  • No help before launch. Most tools assume you’ve already decided to test; few tell you upfront that a page can’t support the test at all

“If you run experiments: the best way to avoid repeated significance testing errors is to not test significance repeatedly.”

Evan Miller, How Not To Run an A/B Test

When a separate calculator earns its place

  • Planning. Check traffic and MDE before you build anything. Speero’s MDE-by-week view or a reverse MDE calculator does this in a minute
  • Sense-checking a surprise. If a small change produced a huge lift, rerun the numbers independently and check SRM before you celebrate
  • Metrics outside the tool. When the goal is demo requests in HubSpot or revenue in your warehouse, export the counts and analyse them yourself
  • Tests without a testing tool. Email subject lines, paid ad variants and landing pages split by URL all need a calculator, because there’s no built-in engine

Which should you use?

You don’t have to pick one. The most reliable approach uses each for what it’s good at:

  • Plan with a calculator. Confirm the page has the traffic to detect a realistic lift in a sensible time
  • Launch in your testing tool, and let its stats engine run, ideally with sequential statistics if you can’t resist looking
  • Analyse with your tool’s result, sense-checked in an independent calculator, then use an LLM to explain what it means for your growth goals and what to try next

If you only use one free calculator, make it Speero for planning and ABTestGuide for a quick post-test check. If your programme runs enough tests that statistical mistakes cost real money, Analytics Toolkit is worth the subscription. If you’re an Optimizely customer, plan with Optimizely’s own calculator so your expectations match its engine. And if AI agents analyse your tests, give them a calculator to call, such as ABTestResult’s MCP server, rather than letting them do the maths themselves.

For more on choosing the platform itself, see the best A/B testing tools.

About Growth Method

A calculator tells you whether one test worked. Growth Method keeps track of what every campaign taught you.

Growth Method is the practical way to use AI in marketing. Our agents suggest campaigns from your own data, then help you plan, launch and analyse them against your growth goals, with no AI vendor lock-in, security taken care of and humans in the loop. Connect a calculator such as ABTestResult and the agent analysing a campaign runs the statistics in real code, then explains the result in plain English.

We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results.

Laura Perrott, Colt Technology Services

Get started and make your next A/B test count for more than one result.

Keep reading