Python Requests Proxy Guide: Auth, Rotation, Retries and SOCKS5

To use a proxy with Python requests, pass a proxies dictionary that maps each URL scheme to a proxy URL: requests.get(url, proxies={"http": proxy, "https": proxy}). Everything else in this guide builds on that one argument: credentials, environment variables, sessions, rotation, retries and SOCKS5. The examples use our residential proxy gateway, gw.proxonym.com:8000, but the patterns work with any HTTP proxy. For a minimal copy-paste setup, see the Python requests integration.

Basic setup: the proxies dictionary#

import requests

PROXY = "http://USERNAME:[email protected]:8000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=30)
print(r.json())   # {'ip': '...'}: the exit IP, not yours

The keys are the scheme of the target URL; the values are the proxy URL. The proxy URL keeps the http:// scheme even for HTTPS targets: requests sends a CONNECT to the proxy and runs TLS end to end with the site. Writing https://gw.proxonym.com:8000 tells urllib3 to speak TLS to the proxy itself, which fails with an SSLError such as WRONG_VERSION_NUMBER.

Credentials with special characters#

If your password contains @, : or /, URL-encode it. Otherwise the proxy URL is parsed wrongly and the gateway answers 407.

from urllib.parse import quote

password = quote("p@ss:w/rd", safe="")   # 'p%40ss%3Aw%2Frd'
PROXY = f"http://USERNAME:{password}@gw.proxonym.com:8000"

Environment variables#

The requests library also reads proxies from the environment, which keeps credentials out of your code:

export HTTP_PROXY="http://USERNAME:[email protected]:8000"
export HTTPS_PROXY="http://USERNAME:[email protected]:8000"
export NO_PROXY="localhost,127.0.0.1"

A proxies= argument on a request overrides the environment. One trap: environment variables override session.proxies, so a stray HTTPS_PROXY can silently replace the proxy you set on a session. Pass proxies= on each request, or set session.trust_env = False to ignore the environment (this also ignores .netrc and CA bundle variables).

Rotating and sticky IPs#

On the gateway, targeting and sessions are set in the username. A small helper keeps that readable:

import secrets
import requests

USERNAME, PASSWORD = "USERNAME", "PASSWORD"

def proxy_for(country=None, city=None, session=None, lifetime=None):
    user = USERNAME
    if country:
        user += f"-country-{country}"
    if city:
        user += f"-city-{city}"
    if session:
        user += f"-session-{session}"
        if lifetime:
            user += f"-lifetime-{lifetime}"
    url = f"http://{user}:{PASSWORD}@gw.proxonym.com:8000"
    return {"http": url, "https": url}

# Rotating: no session, so every new connection gets a new IP
for _ in range(3):
    print(requests.get("https://api.ipify.org", proxies=proxy_for("us"), timeout=30).text)

# Sticky: the same New York IP for up to 30 minutes
sticky = proxy_for("us", "new_york", session=secrets.token_hex(4), lifetime=30)
with requests.Session() as s:
    for _ in range(3):
        print(s.get("https://api.ipify.org", proxies=sticky, timeout=30).text)

Two details matter here. First, rotation happens per connection. requests.get() opens a fresh connection on every call, but a Session keeps connections alive, and HTTPS requests that reuse one CONNECT tunnel share its exit IP. Use a session when you want continuity, and plain calls or a Connection: close header when you want strict rotation. Second, give every worker or task its own random session ID; workers that share an ID share an IP. Rotating vs sticky sessions explains how to choose a lifetime, and the documentation lists every parameter, including -state- and -asn-.

Running requests in parallel#

The requests library is synchronous, so use a thread pool for throughput. Each call below opens its own connection and therefore gets its own IP:

from concurrent.futures import ThreadPoolExecutor

urls = [f"https://example.com/product/{i}" for i in range(1, 101)]

def fetch(url):
    r = requests.get(url, proxies=proxy_for("us"), timeout=(10, 60))
    return url, r.status_code

with ThreadPoolExecutor(max_workers=10) as pool:
    for url, status in pool.map(fetch, urls):
        print(status, url)

The gateway allows unlimited concurrent sessions, but the target sees the sum of your traffic. Keep max_workers modest per domain and raise it only while error rates stay flat.

Static IP lists: datacenter and ISP proxies#

Datacenter, dedicated and ISP proxies do not rotate through a gateway. You get a list of IPs and rotate across it yourself. The account API returns the list; ISP and dedicated entries use the ip:port:username:password format:

import itertools

API = "https://api.proxonym.com/v1"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}
lines = requests.get(f"{API}/proxies/isp?format=txt", headers=HEADERS, timeout=30).text.split()

def to_proxies(line):
    ip, port, user, password = line.split(":")
    url = f"http://{user}:{password}@{ip}:{port}"
    return {"http": url, "https": url}

pool = itertools.cycle([to_proxies(line) for line in lines])
r = requests.get("https://api.ipify.org", proxies=next(pool), timeout=30)

Shared datacenter lists contain ip:port entries on HTTP port 3129: prefix them with your username and password, or whitelist your server IP and use them as they are. For account work on ISP IPs, pin each account to one IP instead of cycling.

Verify the exit IP and location#

info = requests.get("https://ipinfo.io/json", proxies=proxy_for("de"), timeout=30).json()
print(info["ip"], info.get("country"), info.get("city"), info.get("org"))

The org field shows the ASN, which confirms a consumer ISP rather than a hosting provider. Geolocation databases disagree at city level, so treat a city mismatch in one database as a hint, not proof.

Retries, backoff and timeouts#

Residential peers occasionally drop, and targets rate-limit. Mount a retry policy on a session so that transient failures are retried with exponential backoff:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=4,
    backoff_factor=1,                       # waits 0, 2, 4, 8 seconds
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
    respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry)

session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)
session.headers["Connection"] = "close"     # new connection, so a new IP, per attempt

r = session.get("https://example.com/catalog?page=2",
                proxies=proxy_for("us"), timeout=(10, 60))

With no session parameter and Connection: close, each retry leaves through a different IP, which is usually what you want after a block. The cost is a new TLS handshake per request; drop the header for sticky workloads.

Always set a timeout, because requests waits forever by default. The tuple is (connect, read). The connect timeout only covers reaching the gateway, so keep it short. The gateway gives up on a target after 60 seconds and returns 504, so a read timeout of about 60 seconds is a sensible ceiling.

SOCKS5 with requests[socks]#

pip install "requests[socks]"

SOCKS = "socks5h://USERNAME-country-de:[email protected]:1080"
r = requests.get("https://api.ipify.org", proxies={"http": SOCKS, "https": SOCKS}, timeout=30)

Use socks5h://, not socks5://. The h makes the proxy resolve hostnames, so DNS lookups do not leak from your machine and geo-aware DNS answers for the exit location. The trade-offs between the two protocols are covered in SOCKS5 vs HTTP proxies.

Common errors and fixes#

For HTTPS URLs, the gateway answers the CONNECT request, so its status codes arrive as exceptions rather than responses. The message contains the code, for example Tunnel connection failed: 407 Proxy Authentication Required; depending on the urllib3 version, the exception is a ProxyError or a ConnectionError. So for HTTPS URLs, a 403 on a response object comes from the target site, not from the gateway.

A CERTIFICATE_VERIFY_FAILED error is almost always local: an outdated certifi bundle or a corporate proxy that intercepts TLS. Fix the trust store instead of setting verify=False, which hides real problems.

Key takeaways#

• Map both http and https to an http:// proxy URL, and URL-encode special characters in the password.
• An explicit proxies= argument beats environment variables, and environment variables beat session.proxies.
• Rotation is per connection: plain calls rotate, kept-alive sessions reuse an IP.
• Set targeting and sticky sessions in the username, with one random session ID per task.
• Retry 429 and 5xx with exponential backoff, and always pass a timeout.
• For SOCKS5, install requests[socks] and use socks5h://.