Scrapy vs BeautifulSoup vs Selenium: Pick the Right Python Scraper

Key takeaways

BeautifulSoup is a parser, Scrapy is a crawling framework, and Selenium is a browser controller. They solve different jobs, not the same one.

For a few static pages, pair Requests with BeautifulSoup. For large multi-page crawls, use Scrapy. For pages that need a browser to render, use Selenium or Playwright.

On static HTML, Scrapy and asynchronous BeautifulSoup run far faster than sequential Selenium, which trades speed for the ability to run JavaScript.

All three tools can route requests through proxies. The setup differs per tool, and each one is shown in code below.

Quick answer: which tool for which job

BeautifulSoup, Scrapy, and Selenium solve three different jobs. BeautifulSoup parses HTML you have already fetched. Scrapy is a full crawling framework for large, multi-page jobs. Selenium drives a real browser, so it can read pages that build their content with JavaScript. Pick by job: parse a few pages with BeautifulSoup, crawl at scale with Scrapy, render a browser page with Selenium.

Most people comparing these three aren't after definitions. The real question is "which one fits my project?" Here's the short version, and the rest of the page backs it up with tested code, our own speed test, and proxy setup for each tool.

The scale column is a rough guide, not a hard limit. A BeautifulSoup script can work through thousands of pages, but you end up writing the plumbing Scrapy already has. Whatever the size of your web data collection job, pick the tool first. The network setup comes second: further down, you'll find code that connects each tool to Proxy-Cheap proxies.

The three-question decision framework

Answer three questions in order.

Does the content need a browser? Check the raw source. If the data is there, use a plain HTTP request. If JavaScript builds it, use Selenium or Playwright. Also check the Network tab for a direct JSON endpoint.

How many pages? For a few dozen, use Requests + BeautifulSoup. For hundreds or thousands, use Scrapy for built-in concurrency, retries, and exports.

Do you need interaction? Infinite scroll, "load more" buttons, and forms require Selenium or Playwright.

One-line map: Static + few pages = BeautifulSoup. Static + many pages = Scrapy. Browser-rendered or interactive = Selenium or Playwright. For browser rendering at scale, combine Scrapy with scrapy-playwright.

Once you've chosen the tool, choose the IP type. See our overview of the types of proxies for how each option works.

What each tool actually is

BeautifulSoup is an HTML/XML parser. The documentation describes it as a library for extracting data from HTML and XML. Often called "BS4," it doesn't fetch pages or run JavaScript. Pair it with Requests or httpx to download HTML, then use CSS selectors or find() to parse it. It's quick to write, but fetching, retries, concurrency, and exports are yours to manage. The current release is 4.15.0.

Scrapy is a complete crawling framework. You define a spider, while Scrapy handles concurrent requests, link following, retries, duplicate filtering, and JSON, CSV, or XML exports. AutoThrottle can adjust crawl speed based on server response times. Its asynchronous engine supports many requests at once without manual threading. Scrapy 2.19.0 requires Python 3.10+.

Selenium is a browser controller. It drives Chrome, Firefox, Edge, or Safari through WebDriver, so JavaScript and dynamic content work like they do in a real browser. It's heavier because each page loads scripts, styles, and images, and browsers consume significant memory. Use Selenium when data appears only after rendering or when you need clicks and form input.

For programmatic proxy management, see our primer on working with a proxy API.

Side-by-side feature comparison

On features, Scrapy leads for scale: it has built-in concurrency, retries, rate control, and data export. BeautifulSoup is a lightweight parser with none of that built in, so you add it yourself. Selenium runs JavaScript and supports clicks and form input, which the other two cannot do without a browser add-on. Memory use is low for BeautifulSoup and Scrapy and high for Selenium.

¹ Retries are on by default. AutoThrottle is built in but off until you enable it. ² Peak memory for the whole 100-page run in our benchmark below, browser processes included.

The matrix shows a clear split. BeautifulSoup wins on simplicity: install one package, write a loop, done. Scrapy wins on scale and built-ins, which is why it suits long, scheduled crawls of public pages. Those crawls pair well with datacenter proxies when throughput matters most. Selenium wins on browser rendering and interaction, and nothing else in the table comes close there.

Scrapy is also well maintained. Scrapy on GitHub shows 64,000+ stars, and the project shipped four releases between May and September 2026.

Speed and scale on static pages

On static HTML, Scrapy and asynchronous BeautifulSoup finish far faster than sequential Selenium, because Selenium launches and drives a full browser for every page while the other two make plain HTTP requests. Selenium trades that speed for the ability to read pages that only appear after JavaScript runs. Measure on your own target pages before you commit to a tool.

How we tested. Each setup collected 2,000 book titles and links from 100 static listing pages built on the books.toscrape.com markup. Our test machine couldn't reach outside sites, so a local server delivered the pages with 100 ms added to every response, standing in for a network round trip. Times cover the whole run, including Python start-up and browser launch. Each figure is the median of three runs on a 2 vCPU, 8 GB Linux machine.

Concurrency did most of the work. Sequential BeautifulSoup spent most of its 13 seconds waiting, and going async cut that to 4.2 seconds with the same parser. Scrapy edged ahead with zero extra effort. We ran it with scrapy runspider, which allows 8 requests per domain at once. A scrapy startproject project ships politer settings (one request per domain, 1-second delay), so it crawls slower until you tune it.

Selenium was about 26 times slower than Scrapy, partly because the browser also fetches every image, stylesheet, and script. Turning images off cut one run to 56 seconds, still far behind plain HTTP. On pages that only render with JavaScript, though, there's no contest: the other two can't read them.

Faster tools move more data per hour. For long, high-volume runs, unlimited bandwidth proxies take per-GB metering out of the picture. They use SOCKS5, which Requests supports through the requests[socks] extra.

Same task, three tools: the code compared

Here's one job done three ways: collect every book title and link from a listing page, then follow the "next" link until the pages run out. We tested each snippet with Python 3.13, beautifulsoup4 4.15.0, Requests 2.34.2, Scrapy 2.19.0, and Selenium 4.50.0.

BeautifulSoup + Requests

python import requests from bs4 import BeautifulSoup from urllib.parse import urljoin url = "https://books.toscrape.com/catalogue/page-1.html" books = [] with requests.Session() as session:    while url:        response = session.get(url, timeout=10)        response.raise_for_status()        soup = BeautifulSoup(response.text, "html.parser")        for link in soup.select("article.product_pod h3 a"):            books.append({"title": link["title"], "url": urljoin(url, link["href"])})        next_link = soup.select_one("li.next a")        url = urljoin(url, next_link["href"]) if next_link else None print(len(books), "books")

It's a plain script. You own the loop, the error handling, and the output format.

Scrapy spider

python import scrapy class BooksSpider(scrapy.Spider):    name = "books"    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]    def parse(self, response):        for link in response.css("article.product_pod h3 a"):            yield {                "title": link.attrib["title"],                "url": response.urljoin(link.attrib["href"]),            }        yield from response.follow_all(css="li.next a", callback=self.parse)

Run it with scrapy runspider books_spider.py -o books.json. Scrapy hands you concurrency, retries, and the JSON export for free.

Selenium

python from selenium import webdriver from selenium.webdriver.common.by import By options = webdriver.ChromeOptions() options.add_argument("--headless=new") driver = webdriver.Chrome(options=options)  # Selenium Manager fetches the driver books = [] try:    driver.get("https://books.toscrape.com/catalogue/page-1.html")    while True:        for link in driver.find_elements(By.CSS_SELECTOR, "article.product_pod h3 a"):            books.append({"title": link.get_attribute("title"),                          "url": link.get_attribute("href")})        next_links = driver.find_elements(By.CSS_SELECTOR, "li.next a")        if not next_links:            break        driver.get(next_links[0].get_attribute("href")) finally:    driver.quit() print(len(books), "books")

Now you're managing a browser: start it, drive it, and always quit it. The Selenium documentation covers waits and element handling for pages that load content after the first render.

Adding proxies to each tool

All three tools support proxies for reliable, location-accurate collection. BeautifulSoup uses Requests' proxies, Scrapy uses request metadata or downloader middleware, and Selenium uses browser options. Choose rotating residential IPs for broad, multi-market collection, datacenter IPs for high-throughput public pages, or static ISP IPs for consistent sessions.

A single server IP can hit rate limits and return the wrong regional content. Using IPs in the target country improves stability and ensures geo-specific data matches what local visitors see.

Get credentials from the Proxy-Cheap dashboard's Setup Credentials generator and store them in environment variables, not code. We tested all three snippets through a local password-protected proxy.

BeautifulSoup + Requests

python import os import requests from bs4 import BeautifulSoup # Copy host, port, username, and password from your Proxy-Cheap dashboard proxy_url = "http://{}:{}@{}:{}".format(    os.environ["PROXY_USER"], os.environ["PROXY_PASS"],    os.environ["PROXY_HOST"], os.environ["PROXY_PORT"], ) proxies = {"http": proxy_url, "https": proxy_url} response = requests.get("https://books.toscrape.com/", proxies=proxies, timeout=15) soup = BeautifulSoup(response.text, "html.parser") print(response.status_code, soup.title.get_text(strip=True))

Scrapy

python import os import scrapy PROXY = "http://{}:{}@{}:{}".format(    os.environ["PROXY_USER"], os.environ["PROXY_PASS"],    os.environ["PROXY_HOST"], os.environ["PROXY_PORT"], ) class BooksProxySpider(scrapy.Spider):    name = "books_proxy"    async def start(self):        yield scrapy.Request(            "https://books.toscrape.com/catalogue/page-1.html",            meta={"proxy": PROXY},        )    def parse(self, response):        for link in response.css("article.product_pod h3 a"):            yield {"title": link.attrib["title"]}        yield from response.follow_all(            css="li.next a", callback=self.parse, meta={"proxy": PROXY}        )

Scrapy's built-in HttpProxyMiddleware reads the proxy meta key and turns the credentials into a Proxy-Authorization header. It also honors the http_proxy and https_proxy environment variables, as the Scrapy documentation explains. With a rotating residential gateway, rotation happens on the gateway side, so one proxy URL is enough. Custom middleware only matters if you cycle through a list of static IPs yourself.

Selenium

python import os from selenium import webdriver from selenium.webdriver.common.by import By options = webdriver.ChromeOptions() options.add_argument("--headless=new") options.add_argument(    f"--proxy-server=http://{os.environ['PROXY_HOST']}:{os.environ['PROXY_PORT']}" ) options.enable_bidi = True  # WebDriver BiDi answers the proxy login driver = webdriver.Chrome(options=options) driver.network.add_auth_handler(os.environ["PROXY_USER"], os.environ["PROXY_PASS"]) try:    # Navigate over BiDi so the auth handler can respond to the proxy challenge    driver.browsing_context.navigate(        context=driver.current_window_handle,        url="https://books.toscrape.com/",        wait="complete",    )    print(driver.title)    print(len(driver.find_elements(By.CSS_SELECTOR, "article.product_pod")), "books") finally:    driver.quit()

Chrome's --proxy-server flag won't take a username and password, so Selenium 4 answers the proxy login over WebDriver BiDi. On IP whitelist plans, drop the auth handler and keep only the flag.

Which IP type fits which job? Rotating residential proxies suit broad market coverage. They cover 180+ locations with country and city targeting, and a new IP per request or a sticky session of up to 30 minutes. Datacenter IPv4 suits throughput on public pages. For stable sessions, use static residential (ISP) IPs, which keep one address for the whole subscription and support IP whitelist login.

Ready to wire it in? Proxy-Cheap runs on pay-as-you-go billing with no monthly commitment, and you can cancel anytime, with payments secured by 256-bit SSL. Start with static and rotating ISP proxies for session-heavy jobs, and check the Proxy-Cheap API docs when you want to manage proxies from code.

Where Playwright fits in 2026

Playwright is a newer Microsoft-maintained browser automation library with native async support and automatic waiting. It's a strong default for new scraping projects, while Selenium remains useful for existing code and broader browser or language coverage. For Scrapy, scrapy-playwright adds browser rendering inside the framework.

Selenium is not obsolete. Selenium 4.50.0 was released in September 2026, while Playwright 1.63 was released the same month, according to the Playwright for Python release notes.

Playwright's locators wait automatically, its async API supports concurrent pages, and it manages compatible browser builds without separate drivers. Selenium supports more languages, including Ruby, and drives real Safari through safaridriver, while Playwright uses its own WebKit build.

With scrapy-playwright, enable browser rendering only on requests that need it using meta={"playwright": True}. This works well for market research data collection where most pages are static.

When to combine tools

These tools aren't rivals so much as layers. Three pairings come up again and again.

Selenium + BeautifulSoup. Let Selenium render the page, then pass driver.page_source to BeautifulSoup(...) for parsing. BeautifulSoup's search API is often quicker to write than Selenium's element calls.

Scrapy + scrapy-playwright. Keep Scrapy's crawling, retries, and export, and switch on browser rendering per request with meta={"playwright": True}.

Scrapy + BeautifulSoup. Let Scrapy crawl, then run messy HTML through BeautifulSoup inside an item pipeline or callback with BeautifulSoup(response.text, "html.parser").

These pairings are common in price monitoring, where product pages mix static and JavaScript-built content. The rule of thumb stays simple: start with the simplest tool that works, and add a browser or a framework only when the job demands it.