How to Run Screaming Frog in the Cloud (The 2026 Way)

Screaming Frog’s own cloud tutorial has you provision a Google Compute Engine VM, install Chrome Remote Desktop, and drive the full GUI over a remote desktop session. It works. It is also a lot of server to babysit for something you run a few times a month.

There is now a simpler way. For this job we reviewed the modern sandbox and cloud-VM landscape, including E2B, Modal, Daytona, Cloudflare Workers and Sandboxes, and Vercel Sandbox. Most fail a crawler’s basic needs: a heavyweight Java desktop app, crawls that run for hours, and a machine that persists between sessions. Cloudflare Sandboxes caps executions at 30 minutes, Vercel at 45 on its base plan, and the isolate platforms cannot run native binaries at all. One option passed everything, then went further.

Fly.io Sprites give you a hardware-isolated Linux microVM with a 100GB filesystem that appears in a couple of seconds, wakes from sleep in about half a second, costs almost nothing while idle, and can snapshot its entire disk in a third of a second. Every Sprite is also addressable from the outside through a REST API and a native MCP endpoint, which means an AI agent can drive the machine directly. Hold on to that last detail, because it is where this guide ends up.

I set one up to run Screaming Frog headless, crawled a site, saved the whole configured machine as a restore point, and let it go to sleep. Total setup time: about 15 minutes. This guide walks through every step, including the two configuration files the official docs never mention.

Why run crawls in the cloud at all?

Three reasons, same as they have always been:

1. Your laptop stays free. No fan noise through a three-hour crawl, no aborted crawl because you closed the lid.
2. You get a machine sized for the job. Dedicated resources and datacentre bandwidth, not whatever your laptop has spare.
3. Crawls can run on a schedule. That is how you get from “I should re-crawl that site” to actually re-crawling it monthly.

Then there are two newer reasons, and they are the interesting ones:

4. Anyone on your team can interrogate the crawl data. A crawl that lives in the cloud is not trapped on one person’s laptop; the whole team can query it through the Screaming Frog MCP server (more on that later).
5. Marketing platforms can use the crawl data. Growth Method plugs into the same crawls to improve your marketing campaigns.

Why not a traditional VM?

The standard approach costs you in three places:

1. You pay for the VM around the clock, crawling or not.
2. You manage a server: updates, disk, security.
3. You need a GUI. Screaming Frog is a desktop app, so most tutorials bolt on a remote desktop just to click buttons.

A Sprite changes all three:

1. Billing is per second of actual use. The machine sleeps automatically when idle, so a parked crawler costs pennies a month in storage.
2. There is no server to maintain. You can throw the machine away and restore it from a snapshot in seconds.
3. You skip the GUI entirely. Screaming Frog has shipped a proper headless CLI for years, and it is exactly what a cloud crawler should run.

What you need

• A Fly.io account with billing enabled (the smallest Sprites plan is fine)
• A Screaming Frog licence (£199/year). The free tier does not unlock CLI mode, so this is required, not optional
• Ten minutes and a terminal

Step 1: Create your cloud machine

Install the Sprites CLI and log in:

curl -fsSL https://sprites.dev/install.sh | sh
sprite login

Create the machine and open a shell on it:

sprite create seo-crawler
sprite console

That second command drops you into a terminal on a fresh Ubuntu machine with 8GB of RAM and 100GB of disk. From here, everything is ordinary Linux.

Step 2: Install Screaming Frog

Check the download page for the current version number, then download and install the Ubuntu package. Version 24.3 was current when I did this:

curl -fL -o /tmp/sf.deb https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_24.3_amd64.deb
sudo apt-get update
sudo apt-get install -y /tmp/sf.deb

The download is 888MB and took eight seconds on the Sprite’s connection. Java comes bundled, so there is nothing else to install.

Step 3: EULA acceptance and your licence

Run Screaming Frog headless right now and it exits immediately. It wants two things it would normally collect through the GUI: EULA acceptance and your licence. Both live in config files:

mkdir -p ~/.ScreamingFrogSEOSpider

echo "eula.accepted=15" >> ~/.ScreamingFrogSEOSpider/spider.config

printf "yourusername\nYOUR-LICENCE-KEY\n" > ~/.ScreamingFrogSEOSpider/licence.txt
chmod 600 ~/.ScreamingFrogSEOSpider/licence.txt

The licence file is your account username on line one and your licence key on line two, exactly as they appear in your Screaming Frog account. Get either wrong and the log says Licence Status: Invalid; get them right and it says Licence Status: Active with your expiry date. (I lost ten minutes to a missing first character in the key. Check both ends of your paste.)

Step 4: Run your first cloud crawl

screamingfrogseospider --headless \
  --crawl https://yoursite.com \
  --export-tabs "Internal:All,Response Codes:Client Error (4xx),Page Titles:All,Meta Description:All" \
  --output-folder ~/crawl \
  --overwrite

A few minutes later, your exports are sitting in ~/crawl as CSVs. Analyse them right on the machine using the MCP server (more on that later), or pull one down to your laptop by redirecting sprite exec output:

sprite exec -- cat ~/crawl/internal_all.csv > internal_all.csv

One thing to know about exports: they are named views over the crawl, not the whole thing. But with the MCP server and database storage mode, exhaustive up-front exporting becomes unnecessary. Crawls persist in Screaming Frog’s internal database, and the MCP tools (sf_export_crawl, sf_generate_report) pull any tab, bulk export, or report from a stored crawl on demand, later. You stop guessing at export time what you’ll need at analysis time. That is a genuinely better workflow than any --export-tabs list, however long.

Step 5: The step that changes everything

Type exit to leave the machine, then save its entire state:

sprite checkpoint create

This snapshots the full disk, Screaming Frog, your licence, the EULA acceptance, everything, in under a second. Fly’s engineering team describes checkpoints as working “like a git restore, not a system restore”, and that is exactly how it feels in practice. You will never do steps 2 and 3 again. Broke something experimenting? sprite restore v1 puts the machine back. Want next month’s crawl? The machine wakes when you connect, exactly as you left it. (The checkpoints docs cover the details, including the five-snapshot history.)

This is the part a traditional VM cannot match. The Screaming Frog tutorial’s Compute Engine instance is a pet you feed monthly. A checkpointed Sprite is a save file.

What does it cost?

While crawling: fractions of a cent per minute. Simon Willison measured a four-hour working session at about 46 cents. While parked: storage only, billed per block actually written, so a few pence a month. Compare the official tutorial’s setup: Screaming Frog note their Compute Engine instance crawled “3.1 million URLs using database storage mode” for under £20, which is impressive, but that machine bills for every hour it exists.

Honest limits

A Sprite gives you 8GB of RAM (16GB on request). In database storage mode on the included disk, that comfortably covers crawls into the hundreds of thousands of URLs, which is nearly every site nearly everyone crawls. If you audit multi-million-URL sites, the big-VM route in the official tutorial still earns its complexity. For the rest of us, it never did.

The whole thing, condensed

# once, ever
sprite create seo-crawler && sprite console

curl -fL -o /tmp/sf.deb https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_<version>_amd64.deb
sudo apt-get update && sudo apt-get install -y /tmp/sf.deb

mkdir -p ~/.ScreamingFrogSEOSpider
echo "eula.accepted=15" >> ~/.ScreamingFrogSEOSpider/spider.config
printf "yourusername\nYOUR-LICENCE-KEY\n" > ~/.ScreamingFrogSEOSpider/licence.txt
chmod 600 ~/.ScreamingFrogSEOSpider/licence.txt

exit
sprite checkpoint create

# every crawl after that
sprite console
screamingfrogseospider --headless --crawl https://yoursite.com --export-tabs "Internal:All" --output-folder ~/crawl

# optional: expose the MCP server so agents can drive it (next section)

Fifteen minutes of setup, once. After that, cloud crawls whenever you want them, from any machine, for pennies. That alone beats the official tutorial. It is also not the best part.

The real payoff: let your AI run your crawls

Here is where this setup stops being a cheaper VM and becomes a crawler your whole team can talk to. Screaming Frog v24 ships a built-in MCP server (MCP is the open standard that lets AI assistants use tools), which means an AI assistant can drive your crawler: start crawls, poll progress, pull any of its 60-plus reports, run Node.js scripts against the results, and query it all through natural language.

The server runs in two modes, and one of them is built for exactly this setup:

1. STDIO mode is headless. Your MCP client launches the Spider itself in CLI mode, and the official docs put it plainly: “The UI will not be visible to the user, it will run headless.” No GUI, no remote desktop, no open ports. This is the mode a cloud crawler wants.
2. Streamable HTTP mode serves a URL. In the app this is a UI-mode feature at localhost:11435/mcp, but the CLI quietly accepts --mcp-streamable-http-server headless too (the user guide does not mention it; it works). One endpoint, many clients: this is what a shared cloud crawler wants.

Two practical notes for either mode: the MCP requires database storage mode (the default) and a paid licence, and MCP crawls ignore the UI’s speed settings, so pass a saved .seospiderconfig to keep polite crawl rates. Rich Voller’s agency guide to the v24 MCP covers that gotcha and more.

Here is the shape of the shared-endpoint setup, whether you type it or your agent does. A two-line Caddyfile forwards the Sprite’s HTTP port to the MCP server, and both processes register as supervised services so the machine can sleep and wake without you:

printf ':8080 {\n  reverse_proxy 127.0.0.1:11435\n}\n' > ~/Caddyfile

/.sprite/bin/sprite-env services create sf-mcp \
  --cmd /usr/bin/screamingfrogseospider \
  --args "--headless,--crawl,https://example.com,--mcp-streamable-http-server" \
  --env "HOME=$HOME"

/.sprite/bin/sprite-env services create gateway \
  --cmd /usr/bin/caddy \
  --args "run,--config,$HOME/Caddyfile,--adapter,caddyfile" \
  --env "HOME=$HOME" --needs sf-mcp --http-port 8080

Give it a minute on first start (Java plus a warm-up crawl), then your MCP endpoint is live at the Sprite’s URL + /mcp, private by default, Bearer org token to connect.

Does opening that port make your crawler public? No. Caddy’s port is open inside the microVM, but a Sprite’s ports are not reachable from the internet directly. The only road in is Fly’s proxy at the Sprite URL, which enforces authentication before forwarding anything inward:

AI client (Claude, ChatGPT, Growth Method)
  └─ Fly proxy (org token required: the actual gate)
      └─ Caddy :8080 (inside the VM, reachable only via the proxy)
          └─ Screaming Frog MCP on 127.0.0.1:11435 (reachable only by Caddy)

Anonymous requests bounce at Fly’s edge without ever touching Caddy; we watched the gateway logs stay silent until an authenticated request came through. Adding the service changed who answers behind the gate, not the gate itself. The one thing that would change exposure is flipping the Sprite URL’s auth to public, so leave it on the private default. From there, “crawl the site and tell me what’s hurting us” is a sentence you type, not a workflow you run.

For over a decade, Screaming Frog has been a specialist’s tool: one licence, one laptop, one person who knows which of the 60-plus reports matters. Put it on an always-available cloud machine with an MCP endpoint and it becomes shared infrastructure. Your content lead can ask which pages have thin titles. Your developer can ask what broke in the migration. Nobody has to learn what a crawl tab is.

And remember the detail from the top of this guide: the machine itself speaks MCP too. A Sprite exposes a native MCP endpoint of its own, so an agent can wake the machine, kick off the crawl, and query the results through Screaming Frog’s MCP server, end to end, with no human in the loop. Every layer of this crawler is AI-addressable, from the machine to the reports.

That is the capability we build on at Growth Method. Agents run technical audits like this one on a schedule, then turn the findings into a prioritised backlog of marketing campaigns: every broken redirect, thin title, and orphaned page becomes a scoped piece of work with an owner, not a row in a CSV nobody opens. The crawl is no longer the deliverable. It is the input.

About Growth Method

Growth Method is the agentic marketing platform for B2B teams: plan your strategy, run more campaigns, and learn what works, all in one place, for people and agents. Connect a cloud crawler like this one alongside your analytics stack, and the audit-to-campaign loop described above runs on its own.

Book a call to see it working with your stack, or apply for early access.