Build a working web crawler in Python step by step, from a requests and BeautifulSoup starter to a Scrapy crawler that scales with proxies. This step by step guide walks through the core loop that every crawler runs, a complete script you can run today, and how to keep collecting cleanly once you move past a few hundred pages.
A crawler is a loop: fetch a page, extract the links, queue the new ones, mark the page visited, and repeat.
A 30-line Python script with requests and BeautifulSoup is enough to crawl a small site.
Scrapy handles the hard parts (scheduling, retries, throttling, deduplication) once you outgrow the basic script.
Crawling at scale means honoring robots.txt and rate limits and spreading requests across rotating residential proxies for reliable, location-accurate collection.
Crawling and scraping are different jobs. A crawler discovers pages: it follows links to map a site or a section of the web. A scraper extracts specific data from a page once you have it. Crawling is about finding pages; scraping is about reading them. Most real projects use both, with the crawler feeding URLs to the scraper.
A web crawler, also called a spider or a bot, is the same kind of program search engines use to discover and index the web, just pointed at a smaller target. Web crawling is the discovery half of a data pipeline; web scraping is the extraction half. The one-sentence version worth remembering: crawling finds pages, scraping reads them.
People build crawlers for plenty of reasons: search indexing, price monitoring, SEO audits, market research, and building datasets from multiple sources. Data scientists and data analysts lean on crawled web data to feed models and dashboards, while developers use it to power data collection workflows across many websites. Whatever the goal, the crawler's job is narrow: collect data from multiple pages by visiting each one, reading the page content enough to find links, and handing the useful URLs to whatever reads them next.
Every crawler runs the same loop. It keeps two structures: a queue of URLs waiting to be visited (the frontier) and a visited set of URLs already fetched. It pops a URL from the queue, downloads the page, extracts the links, drops any it has already seen, adds the rest to the queue, and repeats until the queue is empty or a limit is hit.
That loop is the whole crawling process, and the entire high level design of a crawler. It does not get more complicated as you scale, it just gets more carefully managed. You start with one or more seed URLs (also called the starting URL). The crawler pops the next URL from the front of the queue, sends an HTTP request to download the page, parses the returned HTML to pull every link, and resolves those into absolute URLs. Any link already in the visited set gets dropped so you never process the same page twice; the rest become new URLs at the back of the queue.
The order you pull URLs decides the shape of the crawl. A FIFO queue gives you breadth-first crawling, where the crawler clears every page at the same level before going deeper, which is why most crawlers reach for it: it tends to find important pages early. Swap in a stack and you get depth-first instead. The crawl keeps going until a stop condition fires: an empty queue, a maximum page count, a depth limit, or a same-domain rule that keeps it on one website. Readers who arrived from a "design a web crawler" interview question can treat this loop as the core of that answer too.
You can build a basic crawler with very little. You need Python 3.x, pip to install packages, and a virtual environment to keep those packages isolated from the rest of your system. The starter stack is two libraries. The requests library is the HTTP client that fetches each web page for you. BeautifulSoup is the parser that reads the HTML content and lets you pull links and text out of it. One fetches, one reads.
You will want more tools as your needs grow. Pages that build their content with JavaScript after the initial load need a headless browser such as Playwright or Selenium, because requests only sees the first HTML response. And when you outgrow a hand-written loop, Scrapy gives you a full framework built for scale.
Decide your scope before you write a line: which domain you are crawling, which URL patterns to include or exclude, how deep you will go, and how many pages you will collect. Pinning this down early keeps the crawl focused and your processing time predictable.
One note on what is appropriate to crawl. Stick to publicly available data, read each site's robots.txt and terms of service first, and do not crawl login required pages or other private content without permission. Treat that as responsible practice and basic security hygiene rather than legal advice; it keeps your project on solid footing.
A working crawler needs about thirty lines of Python. Use a list or deque as the queue, a set for visited URLs, requests to download each page, and BeautifulSoup to parse the HTML and pull every link. Keep links on the same domain, skip ones you have already seen, add a short delay between requests, and stop at a page limit you set.
Create a virtual environment and install the two libraries:
bash python -m venv venv source venv/bin/activate pip install requests beautifulsoup4
Use requests.get() to download one page at a time. Set a descriptive user agent (the User-Agent header) so the site knows who is crawling, check the status_code before you trust the response, and pass a timeout so a slow server cannot stall the whole crawl. The requests library sends your HTTP requests and hands back the HTML page.
Hand the HTML to BeautifulSoup, then call find_all("a", href=True) to collect every link on the HTML page. Links are often relative (/about rather than the full address), so resolve each one against the current URL with urljoin from urllib.parse.
Use collections.deque as the queue and a plain set for visited URLs. Before adding a link, check it is on the same domain and not already in the visited set. That same-page check is what stops the crawler looping forever.
Add time.sleep() between requests so you are not hammering the server, cap the run with a MAX_PAGES limit, and set a depth limit if you only want to go so far.
Here is the whole thing as one runnable file. It ties all five steps together rather than leaving you with disconnected snippets:
python import time from collections import deque from urllib.parse import urljoin, urlparse, urldefrag import requests from bs4 import BeautifulSoup START_URL = "https://example.com/" MAX_PAGES = 50 DELAY_SECONDS = 1.0 HEADERS = {"User-Agent": "ExampleCrawler/1.0 (+https://example.com/crawler-info)"} def normalize(url): # Drop the #fragment so /page and /page#section count as one URL. clean_url, _ = urldefrag(url) return clean_url def extract_links(html, base_url, root_netloc): soup = BeautifulSoup(html, "html.parser") links = [] for tag in soup.find_all("a", href=True): absolute = normalize(urljoin(base_url, tag["href"])) if not absolute.startswith(("http://", "https://")): continue if urlparse(absolute).netloc == root_netloc: links.append(absolute) return links def crawl(start_url, max_pages=MAX_PAGES, delay=DELAY_SECONDS): root_netloc = urlparse(start_url).netloc queue = deque([start_url]) visited = set() while queue and len(visited) < max_pages: url = queue.popleft() if url in visited: continue try: response = requests.get(url, headers=HEADERS, timeout=10) except requests.RequestException as error: print("Skipping", url, "->", error) continue visited.add(url) if response.status_code != 200: continue print("Crawled:", url) for link in extract_links(response.text, url, root_netloc): if link not in visited: queue.append(link) time.sleep(delay) return visited if __name__ == "__main__": pages = crawl(START_URL) print("Done. Crawled", len(pages), "pages.")
Want to stay inside one section of a site? Add a path check in extract_links, for example if urlparse(absolute).path.startswith("/docs"), so the crawler only follows links under that path. That single line keeps a focused crawl from wandering across the whole domain.
Once you need more than a few hundred pages, replace the hand-rolled loop with Scrapy. Scrapy is a Python framework that handles the queue, scheduling, retries, concurrency, and throttling for you. You define a Spider with start URLs and a parse method, set crawl rules and delays in settings, and export the results to JSON or CSV with one command. The Scrapy documentation is the reference for every setting and command shown below.
The basic crawler is great for learning and small jobs. You graduate to Scrapy when reliability and speed optimization start to matter and you are crawling many sites at once. Scrapy gives you a built-in scheduler, automatic retries, concurrency control, AutoThrottle, item pipelines, and structured exports without writing any of that plumbing yourself.
Run scrapy startproject to create the project layout, then write a Spider subclass with start_urls and a parse method. Inside parse, use response.follow to queue each link Scrapy has not seen yet:
python import scrapy class DocsSpider(scrapy.Spider): name = "docs" allowed_domains = ["example.com"] start_urls = ["https://example.com/"] def parse(self, response): yield {"url": response.url, "title": response.css("title::text").get()} for href in response.css("a::attr(href)").getall(): yield response.follow(href, callback=self.parse)
Scrapy reads its rules from settings.py. These five settings cover politeness and depth for most crawls:
python ROBOTSTXT_OBEY = True DOWNLOAD_DELAY = 1.0 CONCURRENT_REQUESTS_PER_DOMAIN = 4 AUTOTHROTTLE_ENABLED = True DEPTH_LIMIT = 3
ROBOTSTXT_OBEY honors each site's robots.txt automatically, DOWNLOAD_DELAY spaces out requests, CONCURRENT_REQUESTS_PER_DOMAIN caps how many run at once, AUTOTHROTTLE_ENABLED adapts the delay to the server's response time, and DEPTH_LIMIT stops the crawl from going too deep.
Run the spider and write structured data straight to a file with one flag: scrapy crawl docs -o results.json or -o results.csv. For anything ongoing, item pipelines clean each record and can push it into a database instead.
For very large jobs, distributed systems such as Scrapyd or Scrapy-Redis spread a crawl across multiple machines. These are advanced topics, but the same core loop sits underneath, so treat distributed crawling as a deep dive once the basics run cleanly.
Crawling from a single IP works for a handful of pages, but at volume a target's rate limits will start returning errors instead of data. Routing requests through a pool of proxies spreads them across many IPs, so each one makes fewer requests and your crawler keeps collecting reliably. The right proxy type depends on your scale, the target, and whether you need a consistent identity.
As request volume climbs, many sites apply rate limits per IP address and start returning failures (HTTP 429 and similar) instead of pages. Proxy rotation solves this: spreading the same workload across a pool of IP addresses means each IP sends fewer requests, which keeps collection steady. This is reliability engineering, not a way around a site's rules. You still honor robots.txt, rate limits, and any anti bot measures the site has in place.
Plugging a proxy in is a one-liner. With the requests starter, pass a proxies dict:
python # Paste the host, port, username, and password from your # Proxy-Cheap dashboard credentials generator (Setup Credentials). proxies = { "http": "http://USERNAME:PASSWORD@HOST:PORT", "https": "http://USERNAME:PASSWORD@HOST:PORT", } response = requests.get(url, headers=HEADERS, proxies=proxies, timeout=10)
With Scrapy, set the same value in the request meta (scrapy.Request(url, meta={"proxy": "http://USERNAME:PASSWORD@HOST:PORT"})) or in a downloader middleware. Copy the exact host, port, username, and password from your Proxy-Cheap dashboard credentials generator; the rotating pool then hands your crawler a fresh exit IP based on the country and session settings you choose.
The proxy type to reach for depends on the crawl. This matrix maps common situations to the product that fits:
| Crawl situation | Proxy type to reach for | Why it fits |
|---|---|---|
| Large public catalogs, documentation, high throughput | Datacenter (IPv4 or IPv6) | Fast, cost-efficient capacity for unprotected, publicly available content |
| Location-accurate data, consumer-facing sites, higher trust | Rotating residential | Real ISP-assigned IPs across a wide global footprint, with a new IP per request |
| Long-lived sessions, account-bound or consistent-identity crawls | Static residential (ISP) | A fixed residential IP with datacenter-grade speed |
| The most demanding targets, mobile-first platforms | Rotating mobile | Mobile carrier IPs with strong IP reputation |
| Sustained, very high-volume throughput | Unlimited bandwidth (SOCKS5) | Unmetered SOCKS5 capacity for heavy, continuous jobs |
Proxy-Cheap runs a large residential pool (155M+ IPs at last count; check the live residential proxies page for the current pool size and country coverage), and its pay-as-you-go billing fits crawling work well: prototype on rotating residential with no monthly commitment, then scale onto static products under the same account when your crawler grows.
Start on pay-as-you-go rotating residential proxies, with no monthly commitment, and scale when your crawler does.
Responsible crawling keeps your project stable and your IPs in good standing. Read each site's robots.txt before you crawl and honor its rules and crawl-delay. Identify your crawler with a clear User-Agent. Space out requests, limit concurrency, and crawl during off-peak hours where you can. Collect only publicly available data, and skip anything behind a login unless you have permission.
robots.txt follows the Robots Exclusion Protocol, a plain text file at the root of a domain (for example, https://example.com/robots.txt) that tells crawlers which paths are open and how fast to go. Python's urllib.robotparser reads it for you, so your crawler can check a URL before sending DNS requests and HTTP requests for it. Many sites also publish a crawl delay directive in that file; treat the crawl delay time as a floor, not a suggestion, and wait at least that long between requests to the same site.
To ensure politeness, lean on four habits. Identify your crawler with a descriptive User-Agent and, ideally, a contact URL. Keep concurrency low and add backoff when a server is slow. Crawl during off-peak hours when you can. And collect only publicly available data, leaving login required pages and private content alone unless you have explicit permission. These same habits are what keep large market research data collection projects sustainable over months rather than days. A few strategies like these are the difference between a crawler that runs for a long time and one that does not.
Most crawler problems fall into four buckets: JavaScript-rendered pages that requests cannot see, duplicate or infinite URLs, request failures at volume, and brittle selectors that break when a site changes. The fixes are a headless browser for JS, URL normalization plus a visited set for duplicates, proxy rotation with retry-and-backoff for failures, and resilient parsing with regular maintenance for selector drift.
JavaScript-rendered content. Plain requests only fetches the initial HTML, so pages that execute JavaScript to load data will look empty. For those JavaScript rendered pages, drive a real browser instead: a headless browser like Playwright, or headless Chrome through Selenium, renders the page so you can read the final page content. Handling dynamic content this way is slower, so reserve it for the pages that actually need it.
Duplicate and infinite URLs. Normalize every URL (strip fragments, sort query parameters) and keep the visited set so the same page never gets fetched twice. Watch for edge cases like pagination traps and calendar links that generate endless new URLs. When the visited set grows too large to hold in memory, a bloom filter gives you fast membership checks at a fraction of the size.
Request failures at volume. Retry with exponential backoff, lower your concurrency, and route through rotating residential proxies so each IP does less work. Frame this as reliability: you are keeping the dynamic content flowing, not working around a site's rules.
Selector drift. Sites change their layout, and a selector that worked last month can return nothing today. Prefer stable selectors, wrap parsing in error handling, and expect ongoing maintenance. Across all four problems the toolkit is the same handful of techniques, and Proxy-Cheap covers the proxy side of it under one account.
