Selenium is a powerful automation tool used for testing, scraping, and browser control. In many cases, you may need to route your browser traffic through a proxy server for anonymity, geo targeting, or managing multiple sessions. Using proxy servers in Selenium allows you to control network traffic and run tests under different network conditions or security setups.
To configure proxies in Chrome, you can use chrome options, which allow you to customize ChromeDriver settings such as proxy configurations and headless mode. Keep in mind that different browsers, like Firefox and Edge, require distinct setup methods for proxy servers and proxy authentication.
This guide explains how to set up a selenium proxy server using Python and Chrome, with clear examples and best practices.
Why Use a Proxy in Selenium
A selenium proxy server allows you to control how your automated browser connects to websites. Common reasons include:
- Changing your IP address
- Avoiding rate limits
- Accessing geo restricted content
- Separating multiple sessions
- Improving anonymity when scraping
- You can add proxies to avoid anti-scraping measures, such as IP blocks and geo-restrictions, and improve the effectiveness of your scraping.
Selenium proxies help mask your IP and simulate users from different regions, allowing for more realistic testing and validation of web content as it appears to users in various locations.
Setting up a proxy in Selenium is straightforward as long as you configure the WebDriver correctly.
How Selenium Handles Proxies
Selenium uses browser specific options to configure proxies, and the process may differ across browsers such as Chrome, Firefox, and Edge. In most cases, you do not set the proxy in Selenium directly. Instead, you pass proxy settings to Chrome, Firefox, or another browser through their capabilities or options, which is essential for effective proxy integration.
The most common setup involves:
- Selenium WebDriver
- A browser driver such as ChromeDriver
- A proxy server with host and port
- Optional username and password for authenticated proxies
To configure selenium for proxy integration, you need to adjust browser-specific options and capabilities to ensure proper proxy handling across different browsers.
Setting Up a Proxy in Selenium Using Python and Chrome
The simplest example is for HTTP or HTTPS proxies without authentication.
Before launching Chrome, you need to configure the proxy using chromeoptions options. In this practical example of proxy configuration, we will launch chrome using the chrome webdriver and create a chromedriver instance with the specified options.
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=http://your-proxy-address:port')
driver = webdriver.Chrome(options=chrome_options)
You can also enable headless chrome by adding the headless mode option to chromeoptions, which allows you to run Chrome without a GUI for automated browsing sessions.
Basic Selenium Proxy Setup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
proxy = "123.45.67.89:8080"
chrome_options = Options()
chrome_options.add_argument(f"--proxy-server=http://{proxy}")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://example.com")
This sends all browser traffic through the specified proxy.
Setting Up an Authenticated Proxy in Selenium
Selenium proxy authentication can be challenging when dealing with proxies that require a username and password. Chrome does not support username and password proxies through command line flags, so you need a small extension or a custom script.
Below is a simple method using a dynamically generated Chrome extension. In this method, you need to specify the proxy url or proxy address (including username and password if required) so that the browser routes its traffic through the desired proxy. The extension parameters should include the proxy server address, which is the location where the proxy is set.
Note: Selenium Wire can be used as an alternative for handling proxy authentication and inspecting or modifying network traffic in Selenium automation workflows.
Create a Proxy Authentication Plugin
First, generate the extension files inside your Python script.
import zipfile
def create_extension(proxy_host, proxy_port, username, password):
manifest = """
{
"version": "1.0.0",
"manifest_version": 2,
"name": "ProxyAuth",
"permissions": ["proxy", "tabs", "unlimitedStorage", "storage", "<all_urls>", "webRequest", "webRequestBlocking"],
"background": {
"scripts": ["background.js"]
}
}
"""
background = f"""
chrome.proxy.settings.set({{
value: {{
mode: "fixed_servers",
rules: {{
singleProxy: {{
scheme: "http",
host: "{proxy_host}",
port: parseInt({proxy_port})
}}
}}
}},
scope: "regular"
}});
chrome.webRequest.onAuthRequired.addListener(
function handler(details) {{
return {{
authCredentials: {{username: "{username}", password: "{password}"}}
}};
}},
{{urls: ["<all_urls>"]}},
["blocking"]
);
"""
with zipfile.ZipFile("proxy_auth.zip", "w") as zp:
zp.writestr("manifest.json", manifest)
zp.writestr("background.js", background)
Then load the extension:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
proxy_host = "123.45.67.89"
proxy_port = "8080"
username = "user123"
password = "pass123"
create_extension(proxy_host, proxy_port, username, password)
chrome_options = Options()
chrome_options.add_extension("proxy_auth.zip")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://example.com")
This method works for most authenticated proxies.
Setting a Proxy for Firefox in Selenium
Configuring proxies in Selenium can vary between different browsers. For example, Firefox offers a direct profile based proxy configuration, which may require different steps compared to Chrome. Each webdriver session can be set up with its own proxy settings, allowing for flexible automation testing and test automation workflows. This is especially useful when you need to control network traffic or test authenticated proxies in Firefox.
Firefox Proxy Example
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "123.45.67.89")
profile.set_preference("network.proxy.http_port", 8080)
profile.set_preference("network.proxy.ssl", "123.45.67.89")
profile.set_preference("network.proxy.ssl_port", 8080)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile, options=options)
driver.get("https://example.com")
Firefox supports proxies more directly than Chrome, making it good for automation.
Setting SOCKS Proxy in Selenium
Selenium also supports SOCKS proxy servers, which are especially useful for tasks requiring high anonymity by routing network traffic through the proxy. You can also use a headless browser with SOCKS proxies to automate tasks without a graphical user interface, making it ideal for web scraping and automated testing scenarios.
SOCKS5 Proxy Example for Chrome
proxy = "123.45.67.89:1080"
chrome_options = Options()
chrome_options.add_argument(f"--proxy-server=socks5://{proxy}")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://example.com")
This will route all traffic through a SOCKS5 proxy.
Proxy Rotation and Management
When running automated tests or web scraping tasks with Selenium, using a single proxy server can quickly lead to your IP address being blocked or flagged by target websites. Proxy rotation is a powerful technique that helps you avoid detection and maintain access by automatically switching between multiple proxies during your Selenium sessions.
Proxy rotation works by assigning a new proxy server to your Selenium WebDriver at regular intervals or after a certain number of requests. This makes your automated traffic appear as if it’s coming from different IP addresses, reducing the risk of bans and helping you bypass rate limits or geo restrictions.
To implement proxy rotation in Selenium, you can maintain a list of proxy addresses and configure your Selenium WebDriver to use a different proxy from the list for each new browser session or request. Here’s a simple Python example demonstrating basic proxy rotation with Selenium:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import random
proxies = [
"123.45.67.89:8080",
"98.76.54.32:3128",
"11.22.33.44:8000"
]
def get_random_proxy():
return random.choice(proxies)
for i in range(5): # Run 5 sessions with different proxies
proxy = get_random_proxy()
chrome_options = Options()
chrome_options.add_argument(f"--proxy-server=http://{proxy}")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://httpbin.org/ip")
print(driver.page_source)
driver.quit()
This approach allows your Selenium scripts to use a different proxy for each session, making it much harder for websites to detect and block your automation. For more advanced proxy management, you can integrate third-party libraries or proxy services that offer automatic proxy rotation, health checks, and support for authenticated proxies.
Testing the Proxy in Selenium
After setting up your selenium proxy server, verify your IP inside the browser:
driver.get("https://httpbin.org/ip")
The script prints the current IP address to the console, allowing you to confirm that the proxy is being used correctly.
Running selenium tests with proxies helps ensure your setup is correct and that your tests will run reliably across different environments.
Check that the returned IP matches your proxy.
Troubleshooting Tips
Browser not using the proxy
Make sure the proxy scheme matches the proxy type:
- http
- https
- socks4
- socks5
Authentication popup appears
Chrome cannot handle authentication directly without the custom extension method.
Proxy connection errors
Check firewall, proxy timeout, or incorrect credentials.
HTTPS sites not loading
Your proxy may only support HTTP or may require CONNECT tunneling.
Summary
A selenium proxy server is easy to configure using Selenium WebDriver and browser options. Using the latest version or latest versions of the selenium package and selenium webdriver manager ensures compatibility and ease of setup. Some tools offer built in support for proxy configuration, streamlining the process for qa teams who need to monitor and troubleshoot network traffic during testing. You can use proxies for anonymity, testing, scraping, or geo targeting. Simulating real user conditions, such as configuring user agents or profiles, is important for accurate testing. In Java, the public static void main method is used as the entry point for selenium scripts. With Python examples for Chrome and Firefox, you can set up both simple and authenticated proxies, including HTTP, HTTPS, and SOCKS.
FAQs
