Web scraping
Collect public data at scale without bans, CAPTCHAs or rate limits.
Use case · AI training data
Feed pretraining corpora, fine-tuning sets and RAG indexes with fresh public web data. Run large crawls on unmetered residential bandwidth and keep costs predictable as your dataset grows into terabytes.
Recommended for AI training data
Pay as you go · Crypto accepted · Live in minutes
The challenge
The problems every team in AI training data runs into sooner or later.
Per-GB pricing works for targeted jobs, but a corpus crawl pulls terabytes. Every recrawl and every new language multiplies the bill, until budget decides how fresh your data can be.
When parts of the web block your crawler, the gaps are not random. Whole regions, languages and site types drop out, and the model quietly learns a skewed picture of the web.
Data teams must show where each document came from, when it was collected and whether the site allowed it. Crawls that ignore robots.txt or site terms create legal and reputational risk.
The solution
Settings you control from the username, with no extra tooling or separate endpoints.
Unlimited residential plans bill by speed, from 100 Mbps to 1 Gbps, for 1, 7 or 30 days. Transfer as much as the line carries without ever watching a GB counter.
Collect from 195+ countries with country, state and city targeting. Local IPs return local-language content, which keeps multilingual datasets balanced instead of English by default.
Run thousands of parallel workers on any plan. Unlimited concurrent sessions and a burst limit around 10,000 requests per second per account keep large crawls moving at full speed.
Residential IPs come from people who opted in through partner apps and can opt out at any time. Your data supply chain starts with consent, which matters to auditors and customers alike.
Recommended
Start with the first one. Most teams add a second pool as their targets grow.
Get started
Estimate pages, average page weight and recrawl frequency. Once you move more than about 1 TB a month and can keep the line busy, unmetered bandwidth tends to cost less than GB packs.
Unlimited plans use a dedicated gateway such as use.unl.proxonym.com, with the same username syntax as residential. Add country targeting per language and keep rotation on.
Write each document with its URL, timestamp and HTTP status, remove near-duplicates and log every robots.txt decision, so each sample in the dataset stays traceable.
Best practices for AI training data
# pip install aiohttp beautifulsoup4
import asyncio
import json
from datetime import datetime, timezone
from urllib import robotparser
from urllib.parse import urljoin, urlsplit
import aiohttp
from bs4 import BeautifulSoup
# Residential gateway. On an unlimited plan use your dedicated gateway instead,
# e.g. use.unl.proxonym.com:8000, with the same username syntax.
PROXY = "http://USERNAME-country-us:PASSWORD@gw.proxonym.com:8000"
AGENT = "ExampleDatasetBot/1.0 (+https://example.com/bot)"
SEED = "https://quotes.toscrape.com/"
robots = {}
async def allowed(session, url):
root = "{0.scheme}://{0.netloc}".format(urlsplit(url))
if root not in robots:
rp = robotparser.RobotFileParser()
try:
async with session.get(root + "/robots.txt", proxy=PROXY) as r:
rp.parse((await r.text()).splitlines() if r.status == 200 else [])
except (aiohttp.ClientError, asyncio.TimeoutError):
rp.disallow_all = True # robots.txt unreachable: stay conservative
robots[root] = rp
return robots[root].can_fetch(AGENT, url)
async def fetch(session, sem, url, out):
async with sem:
if not await allowed(session, url):
return []
try:
async with session.get(url, proxy=PROXY) as r:
if r.status != 200 or "text/html" not in r.headers.get("Content-Type", ""):
return []
html = await r.text(errors="replace")
except (aiohttp.ClientError, asyncio.TimeoutError):
return []
soup = BeautifulSoup(html, "html.parser")
out.write(json.dumps({"url": url, "fetched_at": datetime.now(timezone.utc).isoformat(),
"text": soup.get_text(" ", strip=True)}, ensure_ascii=False) + "\n")
return [urljoin(url, a["href"]).split("#")[0] for a in soup.find_all("a", href=True)]
async def crawl(limit=200, concurrency=16):
sem = asyncio.Semaphore(concurrency)
seen, frontier = {SEED}, [SEED]
timeout = aiohttp.ClientTimeout(total=60)
async with aiohttp.ClientSession(headers={"User-Agent": AGENT}, timeout=timeout) as session:
with open("pages.jsonl", "a", encoding="utf-8") as out:
while frontier:
batch, frontier = frontier, []
for links in await asyncio.gather(*(fetch(session, sem, u, out) for u in batch)):
for link in links:
same_site = urlsplit(link).netloc == urlsplit(SEED).netloc
if same_site and link not in seen and len(seen) < limit:
seen.add(link)
frontier.append(link)
asyncio.run(crawl())FAQ
Still unsure? The help center answers billing, setup and account questions.
Open the help centerLarge crawls run into rate limits and regional blocks that leave gaps in the dataset. Residential proxies spread requests across many consumer IPs and let you collect local-language content from 195+ countries, so coverage reflects the open web rather than what a single data center can reach.
Yes. There is no GB counter: you choose a bandwidth speed from 100 Mbps to 1 Gbps for 1, 7 or 30 days and transfer as much as that speed allows. The pool holds about 10M residential IPs in 180+ countries, and fair use under the acceptable use policy applies.
You pick a dedicated gateway server at checkout: de.unl.proxonym.com in Germany, use.unl.proxonym.com in US East or usw.unl.proxonym.com in US West. Connect on port 8000 for HTTP or 1080 for SOCKS5, using the same username parameters for countries and sessions as on residential.
Collect only public pages, honor robots.txt including rules aimed at AI crawlers, respect site terms and opt-outs, avoid personal data you do not need and keep request rates reasonable. Record provenance for every document so you can answer removal requests and audits.
Yes. Run bulk crawls on an unlimited plan and keep a residential GB pack for occasional jobs on other targets or for countries outside the unlimited pool. Both live in one account with one dashboard, and sub-users keep teams and projects separate.
Platforms
Platform guides with targeting tips, ready-to-run code and the proxy types that work best on each site.
Related
Teams often combine these with AI training data on the same account.
Collect public data at scale without bans, CAPTCHAs or rate limits.
Gather localized content, reviews and trends in any market.
Track competitor prices, stock and reviews across marketplaces.
Pick a plan, pay with crypto and send your first request through Proxonym in minutes.