You know the moment: your scraper runs beautifully on 100 URLs, then falls apart at 10,000. Usually the fix is not more infrastructure, but a handful of habits that keep collection reliable as it scales. This guide walks through the best web scraping practices for 2026, from setting a request budget to reading the new consent signals, so your next big job holds up instead of breaking.
A request budget is the number of requests per minute your scraper may send to one target from one IP. Derive it from four inputs: the rate the target declares, the target's size class, its p95 response latency measured over a 50-request warm-up, and your concurrency cap. Start at the most conservative value the four produce, then raise it only while p95 latency stays flat.
Most guides tell you to add a random delay of one to five seconds. That is a guess, not a budget. The Request Budget swaps the guess for simple arithmetic you can set up in ten minutes, and it is the first of the best web scraping practices because it decides whether your web scraper survives 10,000 URLs. Proxy-Cheap leads with it because infrastructure only helps once your request rate is right.
Four inputs feed the budget, so take them in order:
The rule that ties them together is simple: raise your rate only while p95 latency stays flat. The moment it climbs, you have hit the ceiling, so back off about 30% and stay there. It also helps to schedule your heaviest runs for off-peak hours, which gives the website server more room to breathe.
| Input | How to measure it | Conservative starting value | When to raise it |
|---|---|---|---|
| Declared signal | Read robots.txt and any Retry-After header | Obey the declared value, or 1 request every 2 seconds | Only if no signal is declared |
| Size class | Judge single-server vs CDN vs anti-bot layer | Smallest bucket the evidence supports | When the target is clearly larger |
| p95 latency | 50-request warm-up, record the 95th percentile | Rate that keeps p95 flat | While p95 stays flat |
| Concurrency cap | Count concurrent connections, not just rate | 1 to 2 connections | While failed requests stay near zero |
Consent signals are ranked, not equivalent. Check them in order: a licensed API or bulk download, then Content-Signal directives, then robots.txt under RFC 9309, then the terms of service, then HTTP status codes. Each rung down is weaker evidence that your access is welcome. If you reach rung five and are reading 429s to learn the rules, you skipped four rungs that would have told you first.
Ethical scraping, or responsible web scraping, is not a vibe. It is an order of operations, and it is worth calling the Consent Ladder. Work it from the top down:
A quick word on RFC 9309, since it is where most guides slip. It codified Allow and Disallow rules, the * and $ wildcards, most-specific-match-wins, and case-sensitive path matching. Crucially, crawl-delay was deliberately left out, because real-world behaviour was never consistent enough to standardise. Google ignores crawl-delay outright, Bing treats it as a throttling hint, and Yandex dropped support for it back in 2018. That inconsistency is exactly why you derive your own rate in the section above.
One honest note to close: rungs two and three express preference, not access control, and RFC 9309 says as much. That does not make ignoring them acceptable; it makes honouring them a choice you are accountable for, in line with Proxy-Cheap's appropriate use policy. For the standard itself, read RFC 9309.
Since July 2025, Cloudflare has asked every new domain upfront whether AI crawlers may access its content, and requires crawlers to declare purpose: training, inference, or search. On 1 July 2026 it extended this with a use=reference content signal and set 15 September 2026 as the date new sites and free accounts begin defaulting to allow search while denying training and agent use on pages carrying ads. Cloudflare fronts roughly 20% of the web, so this is a meaningful share of your target list.
Two shifts moved the ground under scrapers, and most guides predate both.
The first, on 1 July 2025, was Cloudflare's permission-based crawler default. New domains are asked upfront, and crawlers are expected to declare whether they are used for training, inference, or search.
The second, on 1 July 2026, added the use content signal and began evolving the pay-per-crawl mechanism toward paying for use rather than each crawl. The compliant syntax a site can now publish is short:
text Content-Signal: search=yes,ai-train=no,use=reference
Per Cloudflare's July 2026 content signals update, on 15 September 2026 new domains onboarding to Cloudflare, including free accounts, begin defaulting to allow search while denying AI training and agent use on pages that display ads.
Alongside this, Web Bot Auth lets crawlers identify themselves cryptographically. So the direction of travel is declared identity, not inferred identity.
The practical takeaway is simple: the web is moving from take-and-apologize to declare-and-negotiate. Scrapers that state their purpose and honor declared preferences will meet fewer 402 and 403 responses than those that do not.
Proxy type should follow the target's checks and your session shape, not your price sensitivity. Datacenter IPv4 delivers high throughput for documentation crawls and unprotected public content. Rotating residential fits geo-specific collection at scale, with country, city and state targeting. Static residential (ISP) fits long paginated sessions where one IP must persist. Mobile fits carrier-specific testing. Pick on fit, then optimise cost.
Two axes decide it. Session shape asks whether one IP must persist across the crawl or should change every request. Target class asks what the target actually checks, from an open documentation site to one behind a WAF. Work those two questions and the proxy types sort themselves.
One honest note on granularity: Proxy-Cheap supports country and ISP targeting on static residential, and country, city and state on rotating residential and mobile. For ASN-specific routing requirements, the rotating residential and mobile lines provide the closest match.
If you are choosing your first line for a data-collection job, start with the data scraping proxies that match your session shape, on pay-as-you-go billing with no monthly commitment.
| Proxy type | Targeting granularity | Session shape | Fits this target class | Billing |
|---|---|---|---|---|
| Datacenter IPv4 | Country | Rotating or fixed | Documentation, sitemaps, unprotected public content | Per-IP monthly |
| Rotating residential | Country, city, state | Changes per request | Geo-specific collection at scale | Pay-as-you-go per GB |
| Static residential (ISP) | Country, ISP | One IP persists | Long paginated sessions | Per-IP monthly |
| Mobile | Country | Sticky or rotating | Carrier-specific testing |
A header set is a claim about which client you are. Make the claim true and internally consistent, because consistency across the set matters far more than rotating one field. A rotated user agent paired with a header set that does not match it is a worse signal than no rotation at all. A real iPhone does not send Chrome's Sec-Ch-Ua, so a mismatched set self-reports.
The headers that travel together are the user agent string, Accept-Language, Accept-Encoding, Sec-Ch-Ua, Referer, and Cookie. Send proper user agent headers as a coherent group rather than swapping one value and hoping.
The header set is not the whole picture, though. TLS fingerprinting reads the handshake as a separate claim, so a Python client that declares Chrome in its user agent header is inconsistent at the transport layer no matter how clean the headers look.
Large targets go further still. Anti-bot systems and anti-scraping technologies model user behavior from signals like mouse movements and the declared operating system, and increasingly apply machine learning to score each request.
Two more hygiene points close it out. First, do not follow honeypot links, the hidden links a real browser never renders, because following one marks your crawler as automated in the least useful way. Second, identify yourself where you can: RFC 9309 recommends the product token appear as a substring of the user agent, and that the identification string describe the crawler's purpose. A named crawler with a contact URL is treated differently from an unnamed one, and for a lot of legitimate collection, declaring who you are works better than staying quiet.
A 429 is not a failure, it is the server publishing its budget. If it carries a Retry-After header, that value is authoritative: wait exactly that long. If it does not, back off exponentially with jitter, starting at two seconds and doubling to a 60 second cap. A 403 is a different message: it means the request itself was rejected, so retrying the same request faster will not help. A 402 means the content is available for payment.
Treat the three as distinct messages, not three flavours of the same failure.
Finally, feed every observed retry value back into your Request Budget, because the framework is a control system, not a fire-and-forget setting. Here is the retry logic in Python, routed through the Proxy-Cheap rotating residential gateway:
python import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry # Rotating residential authenticates with username and password PROXY = "http://USERNAME:[email protected]:5959" # Honour Retry-After, add jitter, cap the wait at 60 seconds retry = Retry( total=5, backoff_factor=2, backoff_max=60, backoff_jitter=1.0, status_forcelist=[429, 500, 502, 503, 504], allowed_methods=["GET", "HEAD"], respect_retry_after_header=True, ) session = requests.Session() adapter = HTTPAdapter(max_retries=retry) session.mount("https://", adapter) session.mount("http://", adapter) proxies = {"http": PROXY, "https": PROXY} response = session.get("https://httpbin.org/status/429", proxies=proxies, timeout=30) print(response.status_code)
For asynchronous requests at higher concurrency, see how to rotate proxies in Python with Requests and AIOHTTP, which fires many API requests without waiting on each response, and pair long retry-heavy sessions with ISP proxies so one IP persists across the run instead of forcing you to rotate IPs frequently.
Caching every response at the HTTP layer is non-negotiable. Re-runs then cost nothing, and you can continuously parse the raw HTML for fields you did not know you wanted the first time, without sending unnecessary requests. Use ETag and If-Modified-Since so the target can answer with a 304 Not Modified, which costs it almost nothing and costs you almost no bandwidth. This is the single practice that most reduces the load your budget has to cover.
Deduplicate on canonical URLs, because multiple URLs resolving to the same content waste budget twice, once on the request and once on the duplicate row.
Re-fetching unchanged pages is the largest avoidable waste in most scrapers. Cloudflare says more than half of crawl traffic from good bots re-fetches pages that have not changed, so your cache is not just performance, it is the cheapest politeness you can buy.
Two habits keep the whole system honest. First, log every request with URL, status, latency, bytes, IP used, and timestamp, because without that log the p95 measurement from the Request Budget is impossible. Second, monitor data shape, not just pipeline health: run data validation on the parsed data, because a scraper returning 200s with empty or misleading data fails silently, which is worse than the excessive requests you can actually see.
Web scraping, sometimes called screen scraping, is the automated extraction of specific fields from web pages into structured data you can store and query. Web scraping helps by retrieving data at scale and providing structured data from target websites that do not offer it, and the program doing the work is a web scraper.
It differs from web crawling in one line: search engines run web crawlers to discover and index pages, while scraping extracts the data points you actually want.
On the tooling side, static HTML yields to an HTTP client and a parser such as BeautifulSoup or a framework like Scrapy. Pages built with JavaScript rendering need headless browsers such as Puppeteer or Playwright, after which you select fields with CSS selectors or XPath, or intercept the XHR calls behind them. The right web scraping tool depends on the site, and a web scraping API is one option when you would rather not maintain the parser yourself.
These best web scraping practices apply to publicly available data. They do not apply to data behind authentication, and Proxy-Cheap's support policy prohibits post-login scraping, so treat anything that requires a login as out of scope. The common data collection activities are price monitoring, SEO and SERP research, ad verification, market research, and localisation testing. For the infrastructure side of those jobs, see the 8 best proxies for web scraping.
It depends on what you collect, where you are, and what you do with it, and honest guidance says so rather than promising a clean yes or no. The legal considerations split into three separate questions the reader usually conflates: is the access lawful, is the data protected, and is the use permitted. Most guides answer only the first.
Publicly available data is the safer ground; data behind authentication is not. Personal data pulls data-protection law such as GDPR into scope regardless of whether the page was public, so the presence of a public URL does not settle it.
Beyond access, two more questions sit apart. Republishing what you collected is a copyright question, distinct from the collection question, and the terms of service are a contract question, distinct again.
None of this is legal advice, and a proxy does not change any of it: a proxy separates IPs, while compliance stays with you. If you want the networking-layer version of the question, see whether are proxies legal covers your case.
