Backing up a scraper is not quite like backing up a normal database, and treating them the same is how teams lose data they can never get back. Some pages you can just re-crawl tomorrow, but a price from last Tuesday is gone for good. This guide covers data backup best practices built for that reality, so you can tier your backups by what each dataset would actually cost to collect again, not by guesswork.
Collected web data is not like a customer database. Some of it can be gathered again, some of it cannot, and the difference decides how often you back it up. A product page can be re-crawled tomorrow. The price that page showed on 14 July is gone forever. Standard backup guidance treats both the same. That is the mistake.
Most data backup best practices were written for production databases and employee laptops, where every byte originates in-house and nothing is re-gatherable. Collected data inverts that: the source of truth is somebody else's website, and it keeps changing under you.
So the two sub-types need naming plainly. State data is what a page says right now and is re-crawlable, while event data is what a page said at a moment in time and is not. When you run large-scale data collection of publicly available data, both land in the same warehouse and look identical.
That is why the useful question is not "how critical is this file" but "what would it cost to get it again." Data loss here is not abstract.
In short, a backup strategy is a data protection and data security decision: which valuable data must stay safe, and which can you simply re-collect? Best practices for data collection and sound data backup strategies start there, and Proxy-Cheap sees it from the collection layer, where the record is captured in the first place.
Re-collection cost is what it would take to gather a dataset again from scratch: bandwidth spend, engineering hours, and the share of records that no longer exist at the source. Tier your backup frequency by that number rather than by a vague sense of criticality. High re-collection cost means back it up more often, keep it longer, and store it immutably.
State the formula once and plainly. Re-collection cost equals volume in GB times your per-GB rate, plus engineer hours times their hourly rate, plus the share of records no longer at the source times their value.
The third term is the one people forget, and it is the one that dominates. Bandwidth is cheap, but vanished records are priceless, because no budget recovers them.
That gives three tiers:
Here is the split in one sketch, with no invented figures. Suppose you hold 40 GB of crawl. Re-collecting it at a pay-as-you-go per-GB rate is a small bill, so as a pure bandwidth cost it is Tier B.
Now suppose 12 percent of those listings have been delisted since capture. That slice is not re-collectable at any bill, which quietly makes part of the same dataset Tier A. The volume did not change, but the re-collection cost did. This is the logic behind reliable price monitoring, where yesterday's price is the whole product.
A collection stack holds four kinds of data, and each has a different re-collection cost. Raw collected data is bulky and partly re-gatherable. Derived datasets rebuild from raw plus code. Configs and credentials are tiny and irreplaceable. Job state and dedupe indices are rebuildable but expensive, because losing them means paying to collect the same records twice.
| Asset class | Typical size | Re-collection cost | Tier | Backup frequency |
|---|---|---|---|---|
| Raw collected data (timestamped) | Large | Infinite for the timestamp, moderate for the content | A | On write |
| Raw collected data (current state) | Large | Bandwidth plus hours | B | Daily |
| Derived and cleaned datasets | Medium | Near zero if raw and code both survive | C | Do not back up, back up the recipe |
| Configs, credentials, endpoint settings | Under 1 MB | Hours of reconstruction, plus downtime | A | On every change |
| Job state, queues, dedupe indices | Small to medium | Duplicate collection spend | B | Daily snapshot |
That is five rows against four classes, because raw data splits by timestamp, and that split is the whole point: the same table can be Tier A and Tier B at once.
The dedupe index earns its own line. Lose it and the next run re-collects everything it already has, which is a direct bandwidth bill and the most commonly missed item on this list. If your collection runs on unlimited bandwidth proxies, a lost index costs hours instead of dollars, but the duplicate work is still real.
Credentials get their own attention too. Proxy usernames and passwords, API keys, IP whitelist entries, rotation intervals, and sticky session settings are all kilobytes of critical data.
Losing them stops the pipeline dead, and no amount of raw HTML brings it back. So treat them as Tier A even though they are the smallest files you own, whether you are running market research workflows or a nightly price crawl.
Raw data goes to compressed object storage with a write-once retention policy. Derived datasets do not need backing up at all if the raw data and the transformation code are both safe. Configs and credentials belong in a secrets manager plus version control, encrypted, saved on every change. Job state gets a daily snapshot alongside the raw data it tracks.
Raw data belongs in compressed object storage with a write-once retention window, so nothing already captured can be altered. Compress before upload and partition by capture date, so a partial restore is possible without pulling the whole archive.
Derived datasets are the counter-intuitive case: back up the pipeline, not the product. If the transformation is deterministic and the code sits in version control, the derived table is a cache you can rebuild, so cloud storage of the output is optional.
Configs and credentials belong in a secrets manager with encryption at rest and rotation, and only authorized users should hold the keys. Version control the config; never the secret.
Job state, especially the dedupe index, gets a snapshot on the same schedule as the raw partition it corresponds to, or the two drift and the restore is worthless.
For the media itself, choose by tier: local storage and network attached storage for fast restores, cloud storage services from major cloud providers for the off-site copy, and hybrid backups that span both storage media. Incremental backups that copy only changed data keep the daily run cheap.
One honest note on cost: re-collection is billed by the gigabyte, which is what makes pay-as-you-go the honest way to price a recovery, with no monthly commitment on the traffic you do not use.
For session-bound collection that must hold one identity, static residential proxies on the static residential (ISP) line keep a fixed IP, while rotating residential proxies suit high-volume state data across markets. The Proxy-Cheap API documentation covers how to script credential rotation so the backup and the live config never disagree.
A backup you have never restored is a guess. Veeam's Data Trust and Resilience Report 2026 found 90 percent of organisations were confident they could recover from a cyber incident, while only 28 percent of ransomware victims fully restored their data. Schedule a restore drill: pick a random backup, rebuild the dataset in a clean environment, and time it.
The confidence gap is the story. Almost everyone believes their backups work, yet fewer than a third of victims got everything back.
For a collection stack, a drill is specific: restore a raw partition, re-run the pipeline against it, and diff the output against the live derived table. If the diff is clean, the backup is real.
Treat that drill as quality assurance for your recovery process, and record the wall-clock time, because that number is your true recovery time objective, not the one in your head. Run it monthly for Tier A and quarterly for Tier B.
A drill also catches silent failures that a green dashboard hides, whether the threat is human error, system failure, or data breaches. It is the core of any disaster recovery plan and any effective backup strategy that claims to protect business continuity, and datacenter proxies make the clean-environment re-crawl cheap when the drill needs live pages.
The 3-2-1 backup rule says keep three copies of your data, on two different media types, with one copy off-site. It comes from photographer Peter Krogh's The DAM Book in 2009 and is documented by CISA. It is a sound floor. It does not tell you which of your assets deserve which tier, which is what re-collection cost is for.
Origin matters, and nobody in this space mentions it. The rule was written for photographers protecting irreplaceable image files, then adopted wholesale by enterprise IT, and CISA documents the Krogh attribution directly.
The irony is worth one line: photographers had exactly the problem collection operators have, because a photograph of a moment cannot be re-taken any more than a delisted price can be re-crawled. That is why the rule runs stricter than most business data needs and looser than Tier A needs.
The common extension, 3-2-1-1-0, adds one copy kept offline or immutable and zero errors on recovery verification, which is simply the restore drill written into the rule.
Two different media might mean local storage plus a cloud region, or network attached storage plus object storage in another data centre, so a lost internet connection or a failed drive on your primary systems never takes every copy at once. The point is to maintain data you can actually restore.
So apply 3-2-1 as the floor for every asset, then apply re-collection cost to decide what climbs above it. Match the ISP proxies or product line to the collection job, and back up the credentials that run it on every change.
