How to Scrape Google Search Results: DIY Proxies vs a SERP API

You can scrape Google search results in two ways: run your own scraper through rotating residential proxies and parse the HTML yourself, or call a SERP API that returns the results as structured JSON. DIY gives you full control but means handling JavaScript rendering, blocks and constantly changing markup. A SERP API turns each results page into one request at a fixed price. Either way, results depend on location, language and device, so those parameters matter as much as the scraping itself.

Why Google is hard to scrape#

• Aggressive rate limiting. Automated patterns get redirected to a /sorry/ page with a CAPTCHA, or receive a 429, and that can start after a small number of rapid queries from one IP.
• JavaScript required. In January 2025 Google began requiring JavaScript to show search results, so plain HTTP clients often get a page asking them to enable it. Most DIY scrapers now need a headless browser.
• Unstable markup. Class names are generated and change often, and parsers built on CSS selectors break without warning.
• Localized results. The same query returns different rankings by country, city, language and device. A scraper that ignores location collects data no real searcher sees.
• Ten results per page. In September 2025 Google stopped honoring the num=100 parameter, so the top 100 positions for a keyword now take ten requests instead of one.

DIY scraping vs a SERP API#

DIY with residential proxies#

A DIY setup has four parts: a headless browser or HTTP client, rotating residential IPs targeted to the searcher's location, a parser for the results page, and logic that detects blocks and consent pages. Residential IPs matter because Google treats hosting ranges with suspicion. City targeting, such as -country-us-city-chicago in the gateway username, makes the IP agree with the location you are measuring. Use a new IP per keyword, and a short sticky session when you paginate one query, so that pages 1 to 3 come from the same visitor.

The cost is mostly traffic plus engineering time. Rendered results pages are heavy. Assume 500 KB of traffic per page with images blocked: 1,000 pages come to 0.5 GB, or $0.37 to $1.00 of residential traffic depending on pack size ($0.74 to $1.99 per GB). That is before retries, browser compute and parser maintenance.

SERP API#

Our Google SERP API takes the query, location, language and device as parameters and returns parsed JSON. You send a query and get structured results back; fetching and parsing happen on our side. One request equals one results page, from $28.70 a month for 100,000 requests (about $0.29 per 1,000) down to $0.16 per 1,000 on the 10 million plan. All plans are on the pricing page.

Example: querying the SERP API#

curl -G "https://serp.proxonym.com/v1/search" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "q=espresso machine repair" \
  --data-urlencode "location=Chicago,Illinois" \
  -d gl=us -d hl=en -d device=desktop -d page=1

The same request in Python:

import requests

resp = requests.get(
    "https://serp.proxonym.com/v1/search",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={"q": "espresso machine repair", "gl": "us", "hl": "en",
            "location": "Chicago,Illinois", "device": "desktop", "page": 1},
    timeout=60,
)
resp.raise_for_status()
data = resp.json()

for item in data["organic_results"]:
    print(item["position"], item["title"], item["link"])
print(len(data.get("ads", [])), "ads,", len(data.get("local_results", [])), "local results")

The response contains these fields:

Localization: gl, hl and location#

• gl is the country the search runs for, as a two-letter code such as us or de. It decides which country's rankings you see.
• hl is the interface language, such as en or de. It changes labels and can shift which language versions of pages rank.
• location narrows the search to a city or region, such as Chicago,Illinois. Local packs and many commercial queries differ completely from one city to the next.

In a DIY scraper, Google infers location from the IP, and the uule URL parameter can encode a more precise location. Whatever you use, keep the signals aligned: a Chicago IP with gl=us and hl=en. Mixed signals, such as a German IP with gl=us, produce blended results that match no real searcher.

Pagination, devices and request rates#

Pagination. Each page is one request: page=1, page=2 and so on in the API, or start=0, start=10 in Google URLs. Decide how deep you really need to track. For rank tracking, the top 30 positions for 1,000 keywords once a day is 3 pages × 1,000 keywords × 30 days = 90,000 requests a month, which fits the 100,000-request plan.

Devices. Mobile and desktop results differ in layout, features and sometimes rankings. Set device=mobile or device=desktop in the API; in a DIY scraper, use a matching user agent and viewport. Track both if most of your audience searches on phones.

Rates for DIY. Rotate IPs per keyword, spread queries evenly over time with random jitter, and avoid bursts at the top of the hour when scheduled jobs tend to start. Detect /sorry/ redirects, 429 responses and consent interstitials, and treat a rising share of them as a signal to slow down rather than to retry harder. A simple check covers the common cases:

def is_blocked(final_url, status, html):
    return (
        status == 429
        or "/sorry/" in final_url
        or "unusual traffic" in html.lower()
    )

Count every blocked response per hour and per exit country, and cut the rate when the share climbs. Do not route queries through CAPTCHA-solving services; lower the rate instead. The general techniques are in web scraping without getting blocked.

Caching. Rankings move daily, not by the minute. Store each keyword's results with a date and reuse them for the rest of the day; a dashboard refresh should never trigger a new query.

Legal considerations#

Collecting publicly visible search results is common practice in SEO and market research, but it is not without rules. Google's terms of service restrict automated access, and its robots.txt disallows crawling of search pages. Whether that creates liability depends on your jurisdiction and on how you use the data. Collect only what you need, avoid storing personal data that appears in results, do not republish copyrighted snippets wholesale, and get legal advice for commercial projects. Our acceptable use policy applies to every request, whichever approach you choose.

Key takeaways#

• DIY Google scraping needs residential IPs, a headless browser, a parser and block handling, so budget for ongoing maintenance.
• A SERP API returns parsed JSON at one request per results page, from $0.16 per 1,000 requests.
• Align gl, hl, location and IP, or you will measure results no real searcher sees.
• Google shows about ten organic results per page, so plan your depth: the top 100 costs ten requests per keyword.
• Throttle, back off instead of solving CAPTCHAs, and respect the legal limits of your use case.