NewUnlimited residential proxies: unmetered traffic on a fixed price. From $33.25/day

Use case · AI training data

Proxies for AI training data at web scale

Feed pretraining corpora, fine-tuning sets and RAG indexes with fresh public web data. Run large crawls on unmetered residential bandwidth and keep costs predictable as your dataset grows into terabytes.

  • 1 GbpsUnmetered bandwidth
  • ~10MIPs on unlimited plans
  • $33.25Per day, no GB meter

Recommended for AI training data

Get started

Pay as you go · Crypto accepted · Live in minutes

The challenge

What gets in the way.

The problems every team in AI training data runs into sooner or later.

  1. 01

    Metered traffic at terabyte scale

    Per-GB pricing works for targeted jobs, but a corpus crawl pulls terabytes. Every recrawl and every new language multiplies the bill, until budget decides how fresh your data can be.

  2. 02

    Blocks distort the dataset

    When parts of the web block your crawler, the gaps are not random. Whole regions, languages and site types drop out, and the model quietly learns a skewed picture of the web.

  3. 03

    Provenance and compliance

    Data teams must show where each document came from, when it was collected and whether the site allowed it. Crawls that ignore robots.txt or site terms create legal and reputational risk.

The solution

How Proxonym solves it.

Settings you control from the username, with no extra tooling or separate endpoints.

Unmetered residential bandwidth

Unlimited residential plans bill by speed, from 100 Mbps to 1 Gbps, for 1, 7 or 30 days. Transfer as much as the line carries without ever watching a GB counter.

Every language and region

Collect from 195+ countries with country, state and city targeting. Local IPs return local-language content, which keeps multilingual datasets balanced instead of English by default.

Throughput without concurrency caps

Run thousands of parallel workers on any plan. Unlimited concurrent sessions and a burst limit around 10,000 requests per second per account keep large crawls moving at full speed.

Ethically sourced IPs

Residential IPs come from people who opted in through partner apps and can opt out at any time. Your data supply chain starts with consent, which matters to auditors and customers alike.

Get started

Up and running in three steps.

  1. 1

    Size the crawl

    Estimate pages, average page weight and recrawl frequency. Once you move more than about 1 TB a month and can keep the line busy, unmetered bandwidth tends to cost less than GB packs.

  2. 2

    Point workers at your gateway

    Unlimited plans use a dedicated gateway such as use.unl.proxonym.com, with the same username syntax as residential. Add country targeting per language and keep rotation on.

  3. 3

    Store with provenance

    Write each document with its URL, timestamp and HTTP status, remove near-duplicates and log every robots.txt decision, so each sample in the dataset stays traceable.

Best practices for AI training data

  • Check robots.txt for every host and honor rules aimed at AI crawlers where sites publish them.
  • Identify your crawler with a descriptive User-Agent that includes a contact URL.
  • Deduplicate with content hashes before storage to save disk space and training time.
  • Accept compressed responses and skip images, video and binaries unless your model needs them.
  • Record the source URL, collection date and any license signals for every document.
Async crawler that honors robots.txt and writes JSONL
# pip install aiohttp beautifulsoup4
import asyncio
import json
from datetime import datetime, timezone
from urllib import robotparser
from urllib.parse import urljoin, urlsplit

import aiohttp
from bs4 import BeautifulSoup

# Residential gateway. On an unlimited plan use your dedicated gateway instead,
# e.g. use.unl.proxonym.com:8000, with the same username syntax.
PROXY = "http://USERNAME-country-us:PASSWORD@gw.proxonym.com:8000"
AGENT = "ExampleDatasetBot/1.0 (+https://example.com/bot)"
SEED = "https://quotes.toscrape.com/"
robots = {}


async def allowed(session, url):
    root = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    if root not in robots:
        rp = robotparser.RobotFileParser()
        try:
            async with session.get(root + "/robots.txt", proxy=PROXY) as r:
                rp.parse((await r.text()).splitlines() if r.status == 200 else [])
        except (aiohttp.ClientError, asyncio.TimeoutError):
            rp.disallow_all = True  # robots.txt unreachable: stay conservative
        robots[root] = rp
    return robots[root].can_fetch(AGENT, url)


async def fetch(session, sem, url, out):
    async with sem:
        if not await allowed(session, url):
            return []
        try:
            async with session.get(url, proxy=PROXY) as r:
                if r.status != 200 or "text/html" not in r.headers.get("Content-Type", ""):
                    return []
                html = await r.text(errors="replace")
        except (aiohttp.ClientError, asyncio.TimeoutError):
            return []
    soup = BeautifulSoup(html, "html.parser")
    out.write(json.dumps({"url": url, "fetched_at": datetime.now(timezone.utc).isoformat(),
                          "text": soup.get_text(" ", strip=True)}, ensure_ascii=False) + "\n")
    return [urljoin(url, a["href"]).split("#")[0] for a in soup.find_all("a", href=True)]


async def crawl(limit=200, concurrency=16):
    sem = asyncio.Semaphore(concurrency)
    seen, frontier = {SEED}, [SEED]
    timeout = aiohttp.ClientTimeout(total=60)
    async with aiohttp.ClientSession(headers={"User-Agent": AGENT}, timeout=timeout) as session:
        with open("pages.jsonl", "a", encoding="utf-8") as out:
            while frontier:
                batch, frontier = frontier, []
                for links in await asyncio.gather(*(fetch(session, sem, u, out) for u in batch)):
                    for link in links:
                        same_site = urlsplit(link).netloc == urlsplit(SEED).netloc
                        if same_site and link not in seen and len(seen) < limit:
                            seen.add(link)
                            frontier.append(link)


asyncio.run(crawl())

More code examplesIntegration guides

FAQ

AI training data questions.

Still unsure? The help center answers billing, setup and account questions.

Open the help center
Why use proxies for AI training data collection?

Large crawls run into rate limits and regional blocks that leave gaps in the dataset. Residential proxies spread requests across many consumer IPs and let you collect local-language content from 195+ countries, so coverage reflects the open web rather than what a single data center can reach.

Is unlimited residential really unmetered?

Yes. There is no GB counter: you choose a bandwidth speed from 100 Mbps to 1 Gbps for 1, 7 or 30 days and transfer as much as that speed allows. The pool holds about 10M residential IPs in 180+ countries, and fair use under the acceptable use policy applies.

Which gateway do unlimited plans use?

You pick a dedicated gateway server at checkout: de.unl.proxonym.com in Germany, use.unl.proxonym.com in US East or usw.unl.proxonym.com in US West. Connect on port 8000 for HTTP or 1080 for SOCKS5, using the same username parameters for countries and sessions as on residential.

How do I keep a training data crawl compliant?

Collect only public pages, honor robots.txt including rules aimed at AI crawlers, respect site terms and opt-outs, avoid personal data you do not need and keep request rates reasonable. Record provenance for every document so you can answer removal requests and audits.

Can I combine unlimited and per-GB plans?

Yes. Run bulk crawls on an unlimited plan and keep a residential GB pack for occasional jobs on other targets or for countries outside the unlimited pool. Both live in one account with one dashboard, and sub-users keep teams and projects separate.

Platforms

Proxies for the sites you collect from.

Platform guides with targeting tips, ready-to-run code and the proxy types that work best on each site.

Related

Related use cases.

Teams often combine these with AI training data on the same account.

Web scraping

Collect public data at scale without bans, CAPTCHAs or rate limits.

Market research

Gather localized content, reviews and trends in any market.

One gateway. Millions of identities.

Pick a plan, pay with crypto and send your first request through Proxonym in minutes.

  • Pay as you go
  • Instant activation
  • Crypto accepted
  • No auto-renewal