Data preprocessing is the set of steps that turn raw, messy data into a clean, structured format a model or analysis can use. The core techniques fall into a few groups: data cleaning (missing values, duplicates, outliers), data transformation (normalization, standardization, encoding), data reduction (dimensionality reduction, feature selection), and data integration and splitting. Each fixes a specific problem before the data reaches your model.
Teams collecting large volumes of public web data, for example through a proxy network like Proxy-Cheap, spend much of the pipeline on exactly these steps.
The main data preprocessing techniques fall into four groups. Data cleaning fixes missing values, duplicates, and outliers. Data transformation reshapes values through normalization, standardization, and encoding. Data reduction shrinks the dataset with dimensionality reduction and feature selection. Data integration merges data from multiple sources. A final step, splitting data into training and test sets, keeps evaluation honest.
| Group | What it fixes | Example techniques |
|---|---|---|
| Cleaning | Errors, gaps, noise | Imputation, deduplication, outlier handling |
| Transformation | Incompatible scales and formats | Normalization, standardization, encoding |
| Reduction | Too many or redundant features | PCA, feature selection, sampling |
| Integration | Data spread across sources | Schema matching, entity resolution, deduplication |
| Splitting | Unreliable evaluation, leakage | Train, validation, test split |
Data cleaning is where most of the actual work happens, and it's usually where the biggest quality gains come from. Raw data almost always has missing data points, duplicate records, or values that don't make sense, and none of that gets fixed by picking a better algorithm.
Missing values show up in nearly every real dataset, and how you handle them depends on why they're missing, not on habit.
For numeric columns with a roughly symmetric distribution, the mean keeps imputation simple. If the column is skewed or contains outliers, the median performs better because it isn't dragged down by extreme values. For categorical variables, the mode (the most frequent category) is the standard choice. If only a small number of rows are missing at random, deleting them is often the cleanest option. For more complex cases, KNN or other model-based imputation can estimate a missing value from similar rows, at the cost of more compute.
The rule that matters most: choose the method based on why the value is missing, not by default.
Duplicates come in two forms. Exact duplicates are identical rows, easy to catch with a simple check. Fuzzy duplicates look almost identical but differ in a formatting detail, a typo, or a timestamp. When rows carry volatile fields, such as a "last updated" timestamp, deduplicate on a stable key instead of the full row, or every near-identical entry will look unique.
Two methods cover most cases. The IQR method flags any value below Q1 minus 1.5 times the interquartile range, or above Q3 plus 1.5 times the interquartile range. The Z-score method flags values that are more than roughly 3 standard deviations from the mean.
The fix depends on the cause. A data-entry error usually gets corrected or removed. A genuine extreme value, like a real high-value transaction, is often capped (winsorized) or transformed rather than deleted outright, since removing it can also remove a real signal. Don't auto-delete any flagged row without first checking why it's flagged.
Noisy data (small, random errors in otherwise valid values) responds well to binning, regression smoothing, or a moving average. Equal-width binning groups values into ranges of the same size; equal-frequency binning groups them so each bin holds roughly the same number of data points.
As a worked example, imagine an "age" column with several missing values and a right-skewed distribution due to a handful of much older customers. Filling those gaps with the median, rather than the mean, keeps the imputed values from being pulled toward the outliers. No single method is universally best; the right one depends on the column's structure.
Once the data is clean, transformation makes it suitable for machine learning algorithms: numeric, on a consistent scale, and free of formats that only make sense to a human reader.
Min-max normalization rescales numerical values to a fixed range, usually 0 to 1, using (x minus the minimum) divided by (the maximum minus the minimum). It suits bounded values and distance-based models, where the raw scale of a feature would otherwise dominate the calculation.
Standardization centers a column to a mean of 0 and a variance of 1, using (x minus the mean) divided by the standard deviation. It's the better choice when features are on different units or a model assumes roughly normal inputs.
As a rule of thumb, distance-based models (KNN, K-means, SVM) and gradient-based models (linear and logistic regression, neural networks) benefit from scaling features to the same scale. Tree-based models, including random forests and gradient boosting, generally don't need it, since they split on thresholds rather than distances.
Machine learning models need numbers, not category labels, so categorical data has to be encoded before it becomes training data. One-hot encoding creates a separate binary column per category and works well for nominal variables with low cardinality. Label or ordinal encoding assigns each category an integer and is suitable for variables with a natural order, such as "low, medium, high." For high-cardinality categorical variables, such as thousands of product IDs, target or frequency encoding avoids the exploding column count that one-hot encoding would create.
Discretization bins a continuous variable into ranges, which can make patterns easier for some models to pick up. A log transform compresses a right-skewed column, useful for values like income or transaction size that span several orders of magnitude. Standard implementations of most of the scaling and encoding steps above, including StandardScaler and OneHotEncoder, are documented in the scikit-learn preprocessing module.
Here's a practical example that comes up constantly in price monitoring: a dataset pulled from several market-specific pages might list one product at "$49.99," another at "€45,99," and a third at "¥6,980." Before any model or comparison can use those numbers, they need to be converted to a single currency and rescaled to the same range as the rest of the dataset.
One detail that's easy to get wrong: scalers and encoders should be fit on the training data only, then applied to validation and test data, never fit on the whole dataset before it's split. More on why that matters in the data integration and splitting section below.
Not every column or row earns its place in the final dataset. Data reduction trims a large or overly complex dataset down to what actually helps model performance and speeds up model training.
Principal component analysis (PCA) projects a dataset onto a smaller number of components that capture the most variance, without needing labeled data. Linear discriminant analysis (LDA) instead seeks the projection that best separates known classes, so it requires labels. Both trade some interpretability of the original columns for speed and, often, less overfitting on large datasets.
Feature selection narrows a dataset to the features that matter most, dropping irrelevant data without creating new data. Filter methods score each feature independently using correlation, the chi-square test, or mutual information. Wrapper methods, such as recursive feature elimination, test feature subsets against the model itself. Embedded methods, such as Lasso regression or tree-based feature importances, select features during training.
Keep feature selection and feature extraction straight: selection keeps a subset of the original columns, while extraction (like PCA) combines them into new, derived features. This is related to, but distinct from, feature engineering, which uses domain knowledge to build new predictive features rather than pick among existing ones.
Simple random sampling pulls rows without regard to their class or category. Stratified sampling preserves the original proportions of each class, which matters when one class is much rarer than another. Sampling data this way is also a practical fix when a scraped dataset is too large to process in full during exploratory data analysis or early testing.
Real-world data analysis rarely works from a single source. Data integration and a disciplined train/test split are what keep a multi-source, real-world data project honest.
Combining structured data from multiple sources raises three recurring problems: schema matching (the same field named differently across sources), entity resolution (the same real-world record under different identifiers), and conflict resolution (two sources disagreeing on a value for the same entity). A deduplication pass after merging catches records that survived integration as near-duplicates.
This comes up constantly in market research, combining internal sales data with external pricing feeds, or in aggregating data from multiple sources like fare or listing data pulled from several providers at once.
Splitting data into three sets keeps model evaluation honest. The training set fits the model. The validation set is used to tune hyperparameters and compare candidate models. The test set remains untouched until the end, providing an unbiased assessment of how the model performs on data it has never seen. A 70/15/15 or 80/20 split (train/test only, with cross-validation standing in for a separate validation set) is typical, though the right ratio depends on how much data you have.
Data leakage occurs when information from outside the training set leaks into the model, typically by fitting a scaler, encoder, or imputer on the full dataset before splitting it. Once that happens, the test set is no longer an independent check because it influences the "training" statistics. The fix: split first, fit every transformation on the training data only, then apply those same fitted parameters to the validation and test sets. This is probably the most common preprocessing bug in real data science projects, and it quietly inflates reported model accuracy until the model meets real, unseen data.
For teams collecting web data, raw data doesn't arrive as a tidy export. It often mixes structured fields with unstructured text pulled straight from the page, and it comes with a specific, predictable set of problems that a generic data-science guide rarely mentions.
Rotating IPs, retried requests, and multiple collection sessions can all cause the same underlying record to be re-fetched more than once. Deduplicating on the raw row misses this, because a small difference in timestamp or formatting makes each fetch look unique. Deduplicate on a stable business key instead, such as a product ID, listing ID, or canonical URL, to catch the real duplicate data regardless of how many times a page was fetched across rotating residential proxy sessions.
Prices, dates, and number formats differ across market-specific pages by design: one page uses a period as the decimal separator, another a comma; one writes a date as day/month/year, another as month/day/year. Left unnormalized, these differences quietly corrupt downstream comparisons and aggregations. Normalize early to a single currency, a single date format (ISO 8601 is a safe default), and a single unit of measurement. This also matters wherever data protection regulations govern how customer data, such as pricing and location details, is stored and compared across regions.
Text pulled from different sites doesn't always arrive as clean UTF-8. Mojibake (garbled characters from a mismatched encoding), leftover HTML entities like &, and stray markup tags all need to be normalized before the text is usable. Repairing encoding early avoids compounding the problem later, when a broken character can silently break a join or a deduplication key.
Not every request in a large-scale collection run finishes cleanly. A timeout or a malformed page can leave a row with some fields populated and others blank, in a way that looks like ordinary missing data but isn't. Quarantine these incomplete rows before imputation, because imputing a value into a row that failed for a structural reason, rather than a random gap, can introduce data that doesn't reflect anything real.
Public sites change their layout, add fields, or restructure existing ones over time, and a scraper built for last month's page structure can start silently returning a different schema. Validate the parsed schema (expected columns and types) before merging new data into the existing dataset, to catch drift before it becomes a downstream bug.
"The preprocessing work on scraped data is rarely about the model. It's almost always about catching what changed on the source pages before that mess reaches the data processing pipeline," says a Proxy-Cheap infrastructure engineer.
For teams running this kind of collection at scale, the proxy layer itself is part of getting clean input data in the first place. Datacenter proxies suit high-throughput crawls of public, unprotected pages, while ISP proxies hold a stable session longer, which helps when a workflow depends on consistent, market-specific connectivity across a long-running collection job. Matching the collection method to the target across the different proxy types available is another way to keep the raw data reaching your preprocessing pipeline consistent in the first place.
Preprocessed data means better data quality reaching the model, less bias baked into the training data, and more reliable, reproducible results. Done well, preprocessing is one of the most reliable ways to improve model accuracy, since even a well-chosen model architecture can learn from noise if the underlying input data is inconsistent.
The oft-cited claim that data scientists spend 80% of their time on data preparation traces back to a 2016 CrowdFlower survey of about 80 data scientists, which found 60% of their time going to cleaning and organizing data, and another 19% to collecting it. That statistic is still repeated constantly, but it's a decade old and reflects a self-selected survey sample rather than the field as a whole.
More rigorous, recent industry surveys tell a different story. Anaconda's State of Data Science report found that data scientists spend about 45% of their time preparing data for modeling, with cleaning and organizing alone accounting for a little over a quarter of the average workday. That's still a substantial share of a data project, just not the four-fifths the older number implies.
The trend since then has moved toward automating more of that 45%. Continuous data-quality checks, pipeline observability tools, and real-time streaming architectures are increasingly handling the repetitive parts of preprocessing as data arrives, rather than in a separate batch step run before every model update.
A small set of tools covers most day-to-day data preparation work. Pandas handles cleaning and reshaping tabular, structured data: filtering rows, renaming columns, merging tables, and handling missing data. Scikit-learn provides the scalers, encoders, and imputers for preprocessing in machine learning workflows, along with a Pipeline object that chains those steps with a model so the same transformations apply consistently to new data. NumPy underlies most numeric operations in both libraries. For messy tabular data that needs manual review before entering a script-based pipeline, OpenRefine provides a spreadsheet-like interface for clustering similar values and standardizing formats by hand.
For teams running repeatable, automated collection that feeds directly into this machine learning workflow, pulling data through the Proxy-Cheap API keeps the collection step itself consistent and scriptable, rather than a manual task repeated before every preprocessing run.
Fitting scalers, encoders, or imputers on the full dataset before splitting causes data leakage. Split the data first, then fit every transformation on the training set only.
Dropping every row with missing data points by default throws away real data and can bias the remaining dataset. Check why the values are missing before deciding whether to impute or delete.
One-hot encoding a high-cardinality categorical variable, like a column with thousands of unique IDs, creates an unmanageable number of columns. Target or frequency encoding handles that case better.
Scaling features for a tree-based model wastes a step it doesn't need. Tree-based models split on thresholds, not distances, so scaling can usually be skipped.
Skipping deduplication on scraped data, because the rows don't look identical, lets rotating sessions and retries quietly duplicate records. Dedup on a stable key instead of the raw row.
Ignoring character encoding until it breaks something downstream turns a five-minute fix into a debugging session. Normalize to UTF-8 as an early pipeline step.
Building an undocumented, one-off pipeline makes it impossible to reproduce results or hand the data project to someone else. Keep every preprocessing step scripted and versioned, not run by hand in a notebook.
