Key takeaways:
Screaming Frog is the crawler most technical SEOs reach for first, and the moment you crawl at scale or audit sites you do not own, a proxy stops being optional. This guide covers what the tool does, how to install and run it, and then the part vendors usually rush: configuring, testing, and rotating proxies the right way.
Screaming Frog SEO Spider is a desktop website crawler built for technical SEO audits. You point it at a domain, and it fetches every URL it can reach, then reports status codes, redirects, page titles, meta descriptions, headings, canonical tags, hreflang, duplicate content, broken links, and dozens of other on-page signals. It is made by Screaming Frog, a search marketing agency in the UK, and it has been a standard part of the technical SEO toolkit since 2010.
The application is written in Java and runs on Windows, macOS, and Ubuntu Linux. It installs and runs entirely on your own machine, so your crawl data stays on your computer. The current release is the version 24 series, which added a built-in MCP server so AI assistants can drive crawls in natural language, plus automatic crawl-to-crawl comparison for scheduled jobs.
Screaming Frog is used wherever someone needs a fast, structured read of how a website is built. The most common jobs:
The last two are where proxies matter most. Auditing a competitor's large site, collecting geo-specific data, or scraping public listings at volume all push past what a single IP can do comfortably.
The features that matter most for day-to-day SEO work:
Partly. The lite version is free to download and use with no signup or email capture, and it crawls up to 500 URLs per crawl. That is enough for small sites and quick spot checks.
A paid license costs around £259 per year per user (confirm the current figure on the Screaming Frog pricing page, since it changes). The license lifts the 500-URL cap and enables the features most audits depend on: saving and loading configuration profiles, JavaScript rendering, custom extraction, custom robots.txt, scheduling, the command-line --config option, and the API integrations.
For proxy users, one detail matters: saving a configuration profile needs a license, and a saved profile is how you reuse a proxy setup in scheduled or command-line crawls. The free tier still lets you set a proxy in the interface for manual crawls.
Installation takes a few minutes on any supported operating system.
A first crawl takes seconds to start.
Each row is one URL, with every technical field laid out in columns you can sort and filter. On the free tier the crawl stops at 500 URLs, which is the first sign you may want a license for real audits. Once a crawl runs cleanly against a small site, you are ready to add a proxy for the bigger jobs.
A crawl from your office or server IP is fine for a small site you own. It stops being fine the moment you scale up or audit a site you do not control, because a single IP becomes the limiter. That is when a crawl runs better through a residential proxy network. For how to weigh providers before you commit, the best proxies for web scraping guide compares the field.
Three concrete reasons to put a proxy in front of a serious crawl:
Match the Proxy-Cheap product to the shape of the crawl:
For a shortlist tuned to search work specifically, see the best SEO proxies comparison.
Each Proxy-Cheap product line fits a different shape of crawl. Pick by the target site, not the budget.
A single proxy entry in Screaming Frog sends every crawl request through the Proxy-Cheap gateway, which fetches the target site from a pool IP and returns the data into the SEO Spider.
Screaming Frog has a native proxy setting, so no extra software is needed. The steps:
Generate the username and password in the Proxy-Cheap dashboard under the Setup Credentials panel, and copy them into the two fields exactly. For a fuller walkthrough of credential generation, see the step-by-step guide to residential proxies.
A quick note on endpoint formats. Rotating residential uses the shared hub thehub.proxy-cheap.com on port 8080, with the country chosen in the dashboard credential generator. Static residential, ISP, and datacenter give you a unique IP and port per proxy, which you copy from the dashboard. Before you paste anything into Screaming Frog, test the credentials with a one-line curl command so you know the proxy itself works:
# Rotating residential hub (HTTP):
# thehub.proxy-cheap.com:8080 (country set in the dashboard credential generator)
# A JSON response with an IP means the proxy works.
# A 407 means the username or password is wrong.
curl -x http://thehub.proxy-cheap.com:8080 \
-U "<your-proxycheap-username>:<your-proxycheap-password>" \
https://httpbin.org/ipThree practical notes from real setups:
Here is the detail most guides skip: Screaming Frog uses one proxy at a time and does not rotate IPs on its own. You still get rotation, in one of two ways.
For a high-volume, multi-country audit, the rotating gateway is the cleaner answer because the provider handles rotation for you. For a long session that should keep one identity, a static residential or ISP proxy is the better fit. The trade-offs are laid out in static vs rotating proxies.
Screaming Frog holds one proxy entry. A rotating gateway cycles the exit IP for you; a static product keeps one IP for the session.
Once a proxy works in the interface, you can carry it into automated crawls. The trick is that the proxy lives inside a saved configuration profile, not in the command itself.
Set the proxy in the interface, then save the configuration with File > Configuration > Save As, which writes a .seospiderconfig file. Saving configs needs a license. Scheduled crawls (File > Scheduling) and command-line crawls then load that file, and the proxy travels with it.
On Windows, the command-line executable is ScreamingFrogSEOSpiderCli.exe:
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://example.com ^
--headless ^
--config "C:\configs\proxy.seospiderconfig" ^
--output-folder "C:\crawls" ^
--save-crawl ^
--export-tabs "Internal:All"On macOS and Linux the executable is screamingfrogseospider:
screamingfrogseospider \
--crawl https://example.com \
--headless \
--config ~/configs/proxy.seospiderconfig \
--output-folder ~/crawls \
--save-crawl \
--export-tabs "Internal:All"--headless runs the crawl with no window, --config loads the saved profile with your proxy inside it, and --export-tabs writes the data you want straight to the output folder. For very large scheduled crawls, set database storage mode in the saved config so memory does not run short mid-run.
Most proxy issues in Screaming Frog come down to a handful of causes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Connection error or timeout on every site after enabling the proxy | Wrong host, port, or credentials | Recheck File > Settings > Proxy, test the credentials with the curl command above, and raise the response timeout under Configuration > Spider > Limits for slow targets |
| 407 Proxy Authentication Required | Missing or mistyped credentials | Tick Use Proxy Credentials and re-enter the exact username and password from the dashboard |
| Settings ignored after you change them | The app was not restarted | Click OK, then close and reopen Screaming Frog; proxy changes apply on restart |
| A large crawl starts returning 403 or 429 | Too many requests from one IP, or crawl speed too high | Lower the speed under Configuration > Speed and route through a rotating gateway so requests spread across many IPs |
| Proxy works in a browser but not in Screaming Frog | App not restarted, or a SOCKS-only endpoint in an HTTP field | Restart the app and use the HTTP host and port from Proxy-Cheap |
| Empty pages on JavaScript-heavy sites | Rendering turned off, or the free tier | Enable JavaScript rendering under Configuration > Spider > Rendering (license required); the render engine uses the same proxy |
| The crawl stops at 500 URLs | Free lite version cap | Enter a license to lift the 500-URL cap and enable saved configs |
If a crawl still struggles after this, slow it down further before anything else. A polite request rate solves more proxy headaches than any single setting.
For technical SEO, Screaming Frog is the default crawler for good reason. It is deep, fast, and runs on your own machine. A few honest trade-offs are worth knowing before a big project.
Pros
Cons
The practical takeaway: run Screaming Frog for the audit, and pair it with a value-tier proxy on pay-as-you-go billing. That combination covers large, geo-specific crawls without enterprise-tier cost or overhead.
