The fastest developer path is Python: requests to fetch, BeautifulSoup to parse, pandas to export to CSV.
Yellow Pages results paginate at 30 listings per page, so pagination handling is essential for full city or category coverage.
Routing requests through a pool of proxies distributes them across many IPs, keeping collection stable and consistent on larger, multi-city jobs.
Match the proxy type to the job: datacenter for small crawls, rotating residential for high-volume multi-city work, static residential or ISP for resumable jobs.
To scrape Yellow Pages, send requests to yellowpages.com category result pages, parse each business listing with a library like BeautifulSoup, and page through the results to collect fields such as name, phone, address, and website. Export the dataset to CSV with pandas. For multi-city collection, route requests through proxies so sessions stay reliable while you gather publicly available business data.
Install requests, beautifulsoup4, and pandas. Send a GET request to a Yellow Pages category URL, parse the HTML with BeautifulSoup, extract each listing's name, phone, address, and website, then paginate through results and export the data to CSV with pandas.
Use Python 3.11+ and install the required libraries:
pip install requests beautifulsoup4 pandas
Requests downloads pages, BeautifulSoup parses them, and pandas builds the output table. Send a standard browser User-Agent with each request.
Yellow Pages organizes directories by city and category:
https://www.yellowpages.com/{city-state}/{category}?page={n}
For example, page 2 of Austin plumbers is https://www.yellowpages.com/austin-tx/plumbers?page=2. Avoid /search?search_terms=... URLs because robots.txt disallows /search* paths. City and category directory pages are open to crawlers.
Reuse a requests.Session, set a timeout, and check the status before parsing:
response = session.get(url, headers=HEADERS, proxies=PROXIES, timeout=30) if response.status_code != 200: print(f"Got status {response.status_code} for {url}")
Parse organic listings inside div.search-results.organic so sponsored cards are excluded. Useful selectors include a.business-name, div.phones, div.street-address, div.locality, a.track-visit-website, and div.categories a. See the BeautifulSoup documentation for CSS selectors.
Use a helper for optional fields:
def get_text(card, selector): element = card.select_one(selector) return element.get_text(" ", strip=True) if element else None
Loop through page numbers and stop when a page has no listings. Keep a max_pages limit while testing. Add a random 2- to 5-second delay between requests to keep traffic reasonable.
Store each listing as a dictionary, create a pandas DataFrame, remove duplicate profile URLs, and export with to_csv(). For JSON, use df.to_json("leads.json", orient="records").
Replace the proxy placeholder with your credentials, or set PROXY_URL = None for a small test without a proxy.
import random import time import pandas as pd import requests from bs4 import BeautifulSoup BASE_URL = "https://www.yellowpages.com" HEADERS = { "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" ), "Accept-Language": "en-US,en;q=0.9", } PROXY_URL = "http://USERNAME:PASSWORD@HOST:PORT" PROXIES = {"http": PROXY_URL, "https": PROXY_URL} if PROXY_URL else None RATING_WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5} def fetch_page(session, url): response = session.get(url, headers=HEADERS, proxies=PROXIES, timeout=30) if response.status_code != 200: print(f"Got status {response.status_code} for {url}") return None return response.text def get_text(card, selector): element = card.select_one(selector) return element.get_text(" ", strip=True) if element else None def get_rating(card): element = card.select_one(".ratings .result-rating") if not element: return None classes = element.get("class", []) stars = next((RATING_WORDS[c] for c in classes if c in RATING_WORDS), None) if stars is not None and "half" in classes: stars += 0.5 return stars def parse_listings(html): soup = BeautifulSoup(html, "html.parser") rows = [] for card in soup.select("div.search-results.organic div.result"): name_link = card.select_one("a.business-name") if not name_link: continue website = card.select_one("a.track-visit-website") reviews = get_text(card, ".ratings .count") rows.append({ "name": name_link.get_text(strip=True), "phone": get_text(card, "div.phones"), "street": get_text(card, "div.street-address"), "locality": get_text(card, "div.locality"), "website": website.get("href") if website else None, "categories": ", ".join( a.get_text(strip=True) for a in card.select("div.categories a") ), "rating": get_rating(card), "review_count": int(reviews.strip("()")) if reviews else None, "profile_url": BASE_URL + name_link.get("href", ""), }) return rows def scrape(city, category, max_pages=5): all_rows = [] with requests.Session() as session: for page in range(1, max_pages + 1): url = f"{BASE_URL}/{city}/{category}?page={page}" html = fetch_page(session, url) if html is None: break rows = parse_listings(html) if not rows: print(f"No listings on page {page}, stopping.") break all_rows.extend(rows) print(f"Page {page}: {len(rows)} listings") time.sleep(random.uniform(2, 5)) return all_rows if __name__ == "__main__": data = scrape("austin-tx", "plumbers", max_pages=3) df = pd.DataFrame(data).drop_duplicates(subset="profile_url") df.to_csv("yellowpages_austin_plumbers.csv", index=False) print(f"Saved {len(df)} businesses to yellowpages_austin_plumbers.csv")
Run python yp_scraper.py. The CSV contains nine columns: name, phone, street, locality, website, categories, rating, review_count, and profile_url.
Yellow Pages selectors can change, so inspect a listing and update selectors if fields suddenly become empty. Email addresses usually aren't shown on results pages, so collecting them requires a second pass over profile_url.
Get script credentials from the Proxy-Cheap dashboard. For browser testing, use the Chrome proxy extension. For programmatic proxy management, see the Proxy-Cheap proxy API.
A scraper using one IP can hit rate limits on larger, multi-city jobs. A proxy pool spreads requests across multiple IPs, keeping collection more stable. Pass a proxies dictionary to requests, then rotate through the pool.
The proxies dict. Use http and https keys with a proxy URL containing your credentials:
proxy = "http://USERNAME:PASSWORD@HOST:PORT" proxies = {"http": proxy, "https": proxy} response = requests.get(url, headers=HEADERS, proxies=proxies, timeout=30)
See the requests proxy documentation. Keep credentials out of version-controlled files and load them from environment variables.
Rotating across a pool. The simplest option is rotating residential proxies. Enable Random IP in the dashboard to get a new IP on each request without changing your code.
You can also manage a pool yourself by generating multiple Session IP credentials and rotating between them:
import time from itertools import cycle import requests PROXY_POOL = cycle([ "http://USERNAME:PASSWORD_1@HOST:PORT", "http://USERNAME:PASSWORD_2@HOST:PORT", "http://USERNAME:PASSWORD_3@HOST:PORT", ]) def fetch_page(session, url, retries=3): for attempt in range(1, retries + 1): proxy = next(PROXY_POOL) proxies = {"http": proxy, "https": proxy} try: response = session.get(url, headers=HEADERS, proxies=proxies, timeout=30) if response.status_code == 200: return response.text print(f"Attempt {attempt}: status {response.status_code}") except requests.RequestException as error: print(f"Attempt {attempt}: {error}") time.sleep(5 * attempt) return None
Replace the original fetch_page() with this version. Failed or 429 requests wait before retrying through the next proxy.
Sticky vs rotating sessions. Per-request rotation works well for independent jobs, such as pulling page 1 from many categories. Sticky sessions are better when requests depend on continuity, such as crawling one category in order. Proxy-Cheap Session IPs keep the same IP for about 30 minutes. For longer persistence, residential proxies also offer static IPs.
Keep the delay. Proxies distribute traffic but don't replace pacing. Keep the 2 to 5 second delay and modest concurrency for steadier collection across multiple cities.
Match the proxy type to the job. Datacenter proxies suit small crawls of public listings, rotating residential proxies fit high-volume multi-city collection, static residential and ISP proxies provide a consistent IP for resumable jobs, and mobile proxies suit the most demanding targets.
| Job profile | Recommended Proxy-Cheap product | Why it fits |
|---|---|---|
| Small, one city or category | Datacenter proxies | High throughput, dedicated IPs, and unlimited bandwidth |
| Multi-city, high volume | Rotating residential proxies | Residential pool across 180+ locations, pay-as-you-go billing, unlimited concurrency |
| Resumable jobs, consistent identity | Static residential proxies or ISP proxies | Same IP for the subscription, with country and ISP targeting |
| Hardest targets | Rotating mobile proxies | Carrier-grade IPs from 100+ countries |
| Very large, continuous jobs | Unlimited bandwidth proxies | No per-GB metering for sustained collection |
A simple rule: estimate your page count. Austin plumbers takes about 17 pages, which a few datacenter IPs can handle. Fifty categories across 30 cities can mean thousands of pages, where rotating residential proxies make more sense.
Budget also matters. Rotating residential and mobile proxies use pay-as-you-go GB billing with no monthly commitment or setup costs. Run a few hundred pages first to measure your real bandwidth cost per 1,000 listings.
Static products are billed per IP per month, and static residential includes a 7-day trial. See our guide to the different proxy types for a broader comparison.
For larger jobs, start with a small pay-as-you-go rotating residential plan, test one city, then scale up.
Collect only publicly available business data, keep request rates reasonable, and add delays between requests. Review Yellow Pages' terms and applicable data-protection laws before running large or repeated jobs.
Use public pages only. Don't collect data behind logins.
Respect robots.txt and terms. The scraper avoids /search* paths because they are disallowed by robots.txt. Review the Yellow Pages terms of service, which prohibit bots and scrapers from collecting site data without prior written consent. For commercial or repeated use, get permission or legal advice.
Throttle requests. Keep the 2- to 5-second delay, limit concurrency, and stop if errors increase.
Handle personal data carefully. Business listings can include personal data, especially for sole proprietors. Collect only what you need, secure it, and honor removal requests.
Yellow Pages, operated by Thryv, is a long-running US business directory covering categories such as plumbers, dentists, restaurants, and law firms.
Listings typically include business name, phone, address, city and ZIP, website, categories, rating, review count, and profile URL. Some also include years in business, opening hours, or email addresses.
Results are organized by location and category. Each page shows 30 organic listings, with sponsored cards in a separate container. A header such as "Showing 1-30 of 491" shows the total, helping you estimate the job size.
Structured local business data feeds plenty of everyday workflows:
B2B lead generation. Build call or outreach lists by category and city, for example, every HVAC contractor in the Dallas area.
CRM enrichment. Fill in missing phone numbers and websites for accounts you already have.
Local market research. Count competitors in a category and compare their ratings to spot underserved neighborhoods. Proxy-Cheap's market research page covers proxy setups for this kind of work.
Sales territory planning. Map business density per city to split territories fairly between reps.
Local SEO and citation research. Check that a business's contact details match across directories.
Each one starts with the same CSV the script produces. For projects that pull from several sites, see our overview of web data collection and our sales and e-commerce use cases.
