Scrapy Splash Web Scraping: A Tested 2026 Guide (Setup, Proxies, and When to Use It)

Key takeaways

Scrapy Splash renders JavaScript pages by routing Scrapy requests through the Splash service, then parsing the rendered HTML like any other response.

It runs today only on a matched version stack. The current scrapy-splash release doesn't import on Scrapy 2.16 or later, and with default settings it fails on its first request from Scrapy 2.14 onward. Our tested pins are below.

Routing Splash requests through residential, ISP, or datacenter proxies keeps long crawls reliable and lets you render pages as they appear in a specific market.

For a brand-new browser-automation project, scrapy-playwright is usually the better starting point. Splash is best for maintaining setups that already run on it.

Scrapy Splash pairs the Scrapy crawling framework with Splash, a lightweight scriptable rendering service, so your spiders can collect data from pages that build their content with JavaScript. Scrapy sends a request to the Splash service, Splash loads the page in its browser engine, and the rendered HTML comes back to your parse method. It still works on a matched version stack, and this guide gives you one that runs.

Does Scrapy Splash still work with current Scrapy versions?

Yes, with a matched version stack. Scrapy Splash still renders JavaScript pages and supports Lua scripts, but the current scrapy-splash is incompatible with the newest Scrapy versions. Pin Scrapy, Twisted, scrapy-splash, and Splash to tested versions, and re-test after upgrades.

Running pip install scrapy scrapy-splash today installs Scrapy 2.19.0 and scrapy-splash 0.11.1, but import fails:

ImportError: cannot import name 'canonicalize_url' from 'scrapy.utils.url'

Scrapy 2.16.0 removed the deprecated scrapy.utils.url.canonicalize_url(), which scrapy-splash still imports. See the Scrapy release notes and scrapy-splash issue #332.

There is another compatibility break. Scrapy 2.14.0 removed the private getslot_key() method that scrapy-splash uses for per-domain request handling. Scrapy 2.14 and 2.15 import successfully but fail on the first SplashRequest, making the 2.13 line the newest tested option with default settings.

Splash itself still works. Splash 3.5, released in June 2020, uses an older QtWebKit engine, while scrapy-splash 0.11.1 was released in February 2025. Both remain useful for existing projects, but development is limited. For new work, consider newer browser automation options.

The tested stack: versions that run together

We tested the Scrapy side of this stack on October 5, 2026. That covered the install and settings, every spider and Lua script in this guide, and the exact requests scrapy-splash sends to Splash. Pin these together:

Pin Twisted explicitly. A Twisted release that's too new for your pinned Scrapy can break a crawl in ways that look unrelated, and Scrapy 2.13.4 declares twisted<=25.5.0. Pinning it yourself keeps a later reinstall from surprising you.

bash python3.12 -m venv .venv source .venv/bin/activate pip install "scrapy==2.13.4" "twisted==25.5.0" "scrapy-splash==0.11.1"

One note on hardware: every published Splash image tag is linux/amd64. On ARM machines, such as Apple Silicon Macs, validate the image under emulation before you rely on it.

Install and run Splash with Docker

Splash runs as a Docker container. If Docker isn't on your machine yet, follow the Docker install guide for your OS first. Then pull the exact image tag instead of latest, so your stack stays reproducible:

bash docker pull scrapinghub/splash:3.5.0 docker run -d --name splash \  -p 127.0.0.1:8050:8050 \  --memory=4.5G --restart=unless-stopped \  scrapinghub/splash:3.5.0 \  --maxrss 4000 --dont-log-args lua_source,password,proxy,proxy_pass

Each flag has a job:

-p 127.0.0.1:8050:8050 binds Splash to loopback, so only your own machine can reach it.

--memory and --maxrss cap memory use. Splash keeps an unbounded in-memory cache, so it grows over a long crawl. With --maxrss, Splash exits when it passes the limit, and the restart policy brings it back.

--dont-log-args keeps listed request arguments, such as passwords and proxy URLs, out of the Splash logs.

Confirm it's alive with the health check:

bash curl http://127.0.0.1:8050/_ping

It returns an "ok" status and the memory used so far. Stop the container with docker stop splash when you're done.

A quick security note. The Splash HTTP API accepts any URL and any Lua script from whoever can reach it. Keep it on loopback or behind an access-controlled private network, and never expose port 8050 to the public internet.

Wire scrapy-splash into your Scrapy project

Add these settings to settings.py:

SPLASH_URL = "http://127.0.0.1:8050" DOWNLOADER_MIDDLEWARES = {    "scrapy_splash.SplashCookiesMiddleware": 723,    "scrapy_splash.SplashMiddleware": 725,    "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 810, } SPIDER_MIDDLEWARES = {    "scrapy_splash.SplashDeduplicateArgsMiddleware": 100, } REQUEST_FINGERPRINTER_CLASS = "scrapy_splash.SplashRequestFingerprinter"

The downloader middlewares send SplashRequest calls to Splash and return rendered responses to Scrapy. HttpCompressionMiddleware at 810 lets scrapy-splash process responses correctly. The spider middleware deduplicates repeated arguments, while the fingerprinter makes duplicate filtering and caching aware of Splash arguments.

Skip the older DUPEFILTER_CLASS and HTTPCACHE_STORAGE settings. Current scrapy-splash deprecates them in favor of the fingerprinter.

The flow is: your spider creates a request, scrapy-splash sends it to Splash, Splash loads the page and runs JavaScript, then the rendered HTML returns to your callback. During rendered requests, Scrapy does not connect directly to the target site.

Expect some deprecation warnings because scrapy-splash depends on Scrapy internals that newer versions changed. Keep the stack pinned and re-test upgrades. For automated proxy management, see our guide to the public proxy API.

Render a JavaScript page with SplashRequest

To render a page, yield a SplashRequest instead of a standard Scrapy Request. Point it at your target URL, choose the render.html endpoint, and set a wait so client-side scripts finish before Splash returns the HTML. Your callback then parses the rendered response with the same selectors you would use on any Scrapy response, so only the fetch step changes, not your extraction logic.

First, see the problem. The quotes on https://quotes.toscrape.com/js/ are written into the page by an inline script. A plain scrapy.Request to that URL returns HTML with zero div.quote elements in our test. Now the Splash version:

python import scrapy from scrapy_splash import SplashRequest class QuotesJsSpider(scrapy.Spider):    name = "quotes_js"    def start_requests(self):        yield SplashRequest(            "https://quotes.toscrape.com/js/",            callback=self.parse,            endpoint="render.html",            args={"wait": 2},        )    def parse(self, response):        self.logger.info("Title: %s", response.css("title::text").get())        for quote in response.css("div.quote")[:3]:            yield {                "text": quote.css("span.text::text").get(),                "author": quote.css("small.author::text").get(),            }

Run it with scrapy crawl quotes_js -O quotes.json. The log shows Title: Quotes to Scrape, and the file holds three items:

json [ {"text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”", "author": "Albert Einstein"}, {"text": "“It is our choices, Harry, that show what we truly are, far more than our abilities.”", "author": "J.K. Rowling"}, {"text": "“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”", "author": "Albert Einstein"} ]

Notice the entry point. Since Scrapy 2.13, the recommended entry point is an async start() method, and start_requests() is deprecated. On this stack, though, a spider that defines only async def start() stops with a ValueError. The reason is that SplashDeduplicateArgsMiddleware only implements the older process_start_requests() hook. Use start_requests() as shown, and accept the deprecation warning that comes with it.

The wait value is in seconds and must be lower than the render timeout, which defaults to 30. The same pattern scales to the data-collection workflows you'd run across hundreds of pages.

Run Lua scripts for interaction: wait, scroll, and log in

When a fixed wait isn't enough, switch to the /execute endpoint and send a Lua script. Splash runs the script's main(splash, args) function, where args is the same table as splash.args. Pass values through those args instead of pasting them into the script string. That keeps the script cacheable and keeps secrets out of it.

Wait for an element. This script checks for a selector every half second and returns the HTML once it appears:

python WAIT_FOR_ELEMENT = """ function main(splash, args)  assert(splash:go(args.url))  local found = false  for _ = 1, args.max_checks do    if splash:select(args.selector) then      found = true      break    end    splash:wait(args.check_interval)  end  return {found = found, html = splash:html()} end """ def start_requests(self):    yield SplashRequest(        "https://quotes.toscrape.com/js-delayed/?delay=3000",        callback=self.parse,        endpoint="execute",        args={            "lua_source": WAIT_FOR_ELEMENT,            "selector": "div.quote",            "max_checks": 20,            "check_interval": 0.5,        },    )

In the callback, response.data["found"] tells you whether the element showed up. Selectors like response.css() still work on the returned HTML.

Scroll and count. Infinite-scroll pages load more records when you reach the bottom. This script records the count before and after a few scrolls, so you can check what actually happened:

lua function main(splash, args)  assert(splash:go(args.url))  assert(splash:wait(args.wait))  local count = splash:jsfunc([[    function() { return document.querySelectorAll(".quote").length; }  ]])  local before = count()  for _ = 1, args.scrolls do    splash:runjs("window.scrollTo(0, document.body.scrollHeight);")    splash:wait(args.scroll_wait)  end  return {before = before, after = count(), html = splash:html()} end

Read both numbers before you trust the result. If after equals before, the scroll didn't trigger the page's loader on Splash's engine, and adding more waits rarely fixes that. Many infinite-scroll pages pull records from a JSON endpoint (the quotes sandbox uses /api/quotes?page=N), and calling that endpoint with a plain Scrapy request is faster than rendering anyway.

Fill and submit a login form. The quotes sandbox at /login accepts any username and password, so it's a safe practice target:

lua function main(splash, args)  assert(splash:go(args.url))  assert(splash:wait(1))  local form = assert(splash:select("form"))  assert(form:fill({username = args.username, password = args.password}))  assert(form:submit())  assert(splash:wait(2))  return {    logged_in = splash:select("a[href='/logout']") ~= nil,    html = splash:html(),  } end

Pass the credentials as args read from environment variables, such as os.environ["SITE_USERNAME"]. Never write them into lua_source, logs, or screenshots. Then check the authenticated marker in your callback, and stop if response.data["logged_in"] is false. A 200 response alone doesn't prove the login worked.

Multi-step flows like this need the same IP from the first page to the last. Static residential proxies give you one fixed residential IP for the whole subscription, which suits authenticated sessions.

Route Splash requests through proxies for reliable, location-accurate rendering

Splash makes the outbound request, not Scrapy, so Scrapy's proxy middleware does not apply to rendered fetches. Instead, pass the proxy to Splash through SplashRequest or a Lua script. Residential, ISP, and datacenter IPs can provide location-specific renders and more reliable long crawls.

We tested this with an authenticated proxy. meta={"proxy": ...} only proxied the request from Scrapy to Splash, not the target page. Environment variables such as HTTP_PROXY behave the same way.

Option 1: Proxy argument. For render.html, pass the proxy in args. The Splash HTTP API docs accept protocol://user:password@host:port with HTTP or SOCKS5:

import os def start_requests(self):    proxy_url = os.environ["PROXY_URL"]    yield SplashRequest(        "https://quotes.toscrape.com/js/",        callback=self.parse,        endpoint="render.html",        args={"wait": 2, "proxy": proxy_url, "images": 0},    )

Keep credentials in environment variables and exclude proxy from --dont-log-args so they stay out of Splash logs.

Option 2: Lua with request:set_proxy. Use this with /execute scripts or when different requests need different proxies:

function main(splash, args)  splash:on_request(function(request)    request:set_proxy{      host = args.proxy_host,      port = args.proxy_port,      username = args.proxy_user,      password = args.proxy_pass,      type = "HTTP",    }  end)  assert(splash:go(args.url))  assert(splash:wait(args.wait))  return {html = splash:html()} end

Pass proxy_port as a number. HTTP proxying also works for HTTPS pages through CONNECT.

Two details matter:

Splash sends scripts, styles, and images through the proxy. Set images: 0 to reduce bandwidth.

Per-request IP rotation can give one render multiple IPs. For consistent sessions, use Proxy-Cheap session credentials, which keep an IP for about 30 minutes.

Match the proxy to the workload:

Rotating residential: High-volume rendering across markets. Rotating residential proxies cover 180+ locations with per-request rotation or sticky sessions.

Static residential (ISP): Stable IPs for multi-step or authenticated flows. ISP proxies combine residential identity with datacenter speed.

Datacenter IPv4: High-throughput rendering of public content. Datacenter proxies offer unlimited bandwidth and per-IP pricing.

Mobile: Mobile-first sites and carrier-network testing. Mobile proxies cover 100+ countries.

Proxying also enables geo-specific rendering. Choose a country for the exit IP to see localized prices, languages, or listings as local visitors do.

Proxy-Cheap uses pay-as-you-go billing with no monthly commitment. Start with a small rotating residential plan and compare renders from your server IP and target market.

When to choose Scrapy Splash, scrapy-playwright, or a managed approach

There's no single right answer here. It depends on what you already run and how much infrastructure you want to own.

Maintain Scrapy Splash when you already run it and your targets render correctly on Splash's engine. A working spider on a frozen stack is cheap to keep. Pin the versions above and keep Splash on loopback. Re-test before any upgrade.

Start with scrapy-playwright for new projects. It drives a current Chromium, Firefox, or WebKit build, with modern waits and locators. Version 0.0.48 shipped in July 2026, and the scrapy-playwright project supports the async start() entry point. The proxy lesson carries over, because the browser makes the request. You set proxies in PLAYWRIGHT_LAUNCH_OPTIONS or per context in PLAYWRIGHT_CONTEXTS, not through Scrapy's proxy middleware, and per-request meta["proxy"] isn't supported.

Consider a managed rendering approach when you don't want to run and pin rendering infrastructure at all. You trade control and per-page cost for less maintenance. That can make sense for small teams with many targets and little time for upkeep.

A simple test helps. If a Splash render of your page looks right and your spider is stable, stay put. If pages render incompletely on Splash's older engine, or you're starting fresh, move to scrapy-playwright.

The same choice comes up in SEO research and monitoring, where rendered search and product pages feed daily reports. Whichever tool you choose, pick the proxy by workload. Our overview of proxy types explains the trade-offs.

Troubleshooting common Scrapy Splash errors

Most Scrapy Splash failures trace back to versions, script arguments, or timing. Match the symptom, then apply the fix:

Every fix here is a version pin, a settings change, or more diagnostic time. Don't edit files inside installed packages or patch Scrapy internals. Those changes disappear on the next install and hide the real version mismatch. For anything not listed, search the scrapy-splash issue tracker for your exact error text.