---
title: "How to Scrape Google Search Results at Scale | Proxonym"
description: "How to scrape Google search results: DIY with residential proxies vs a SERP API, localization with gl, hl and location, pagination, devices, rate limits."
url: https://proxonym.com/blog/scrape-google-search-results
updated: 2026-10-01T21:31:28+00:00
---

Guides · October 1, 2026 · 5 min read

# How to Scrape Google Search Results: DIY Proxies vs a SERP API

Why Google is hard to scrape, when to build your own scraper on residential proxies and when to call a SERP API, plus localization, pagination, device and legal notes.

By the Proxonym team

You can scrape Google search results in two ways: run your own scraper through rotating residential proxies and parse the HTML yourself, or call a SERP API that returns the results as structured JSON. DIY gives you full control but means handling JavaScript rendering, blocks and constantly changing markup. A SERP API turns each results page into one request at a fixed price. Either way, results depend on location, language and device, so those parameters matter as much as the scraping itself.

## Why Google is hard to scrape

- **Aggressive rate limiting.** Automated patterns get redirected to a `/sorry/` page with a CAPTCHA, or receive a `429`, and that can start after a small number of rapid queries from one IP.
- **JavaScript required.** In January 2025 Google began requiring JavaScript to show search results, so plain HTTP clients often get a page asking them to enable it. Most DIY scrapers now need a headless browser.
- **Unstable markup.** Class names are generated and change often, and parsers built on CSS selectors break without warning.
- **Localized results.** The same query returns different rankings by country, city, language and device. A scraper that ignores location collects data no real searcher sees.
- **Ten results per page.** In September 2025 Google stopped honoring the `num=100` parameter, so the top 100 positions for a keyword now take ten requests instead of one.

## DIY scraping vs a SERP API

### DIY with residential proxies

A DIY setup has four parts: a headless browser or HTTP client, rotating residential IPs targeted to the searcher's location, a parser for the results page, and logic that detects blocks and consent pages. Residential IPs matter because Google treats hosting ranges with suspicion. City targeting, such as `-country-us-city-chicago` in the gateway username, makes the IP agree with the location you are measuring. Use a new IP per keyword, and a short sticky session when you paginate one query, so that pages 1 to 3 come from the same visitor.

The cost is mostly traffic plus engineering time. Rendered results pages are heavy. Assume 500 KB of traffic per page with images blocked: 1,000 pages come to 0.5 GB, or $0.37 to $1.00 of [residential traffic](https://proxonym.com/residential-proxies) depending on pack size ($0.74 to $1.99 per GB). That is before retries, browser compute and parser maintenance.

### SERP API

Our [Google SERP API](https://proxonym.com/serp-api) takes the query, location, language and device as parameters and returns parsed JSON. You send a query and get structured results back; fetching and parsing happen on our side. One request equals one results page, from $28.70 a month for 100,000 requests (about $0.29 per 1,000) down to $0.16 per 1,000 on the 10 million plan. All plans are on the [pricing page](https://proxonym.com/pricing).

| Factor | DIY with residential proxies | SERP API |
| --- | --- | --- |
| Output | Raw HTML that you parse | Parsed JSON |
| Cost per 1,000 pages | Traffic, plus compute, retries and maintenance | $0.16 to $0.29 |
| Maintenance | Parser updates whenever the markup changes | Parsing maintained on our side |
| Flexibility | Anything visible on the page, screenshots, custom flows | The fields the API returns |
| Good fit | Unusual SERP features, full-page captures, existing scraping stacks | Rank tracking, keyword research, competitor monitoring |

## Example: querying the SERP API

**Shell**

```bash
curl -G "https://serp.proxonym.com/v1/search" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "q=espresso machine repair" \
  --data-urlencode "location=Chicago,Illinois" \
  -d gl=us -d hl=en -d device=desktop -d page=1
```

The same request in Python:

**Python**

```python
import requests

resp = requests.get(
    "https://serp.proxonym.com/v1/search",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={"q": "espresso machine repair", "gl": "us", "hl": "en",
            "location": "Chicago,Illinois", "device": "desktop", "page": 1},
    timeout=60,
)
resp.raise_for_status()
data = resp.json()

for item in data["organic_results"]:
    print(item["position"], item["title"], item["link"])
print(len(data.get("ads", [])), "ads,", len(data.get("local_results", [])), "local results")
```

The response contains these fields:

| Field | Contents |
| --- | --- |
| `search_parameters` | The parameters the search ran with |
| `organic_results[]` | `position`, `title`, `link` and `snippet` for each organic result |
| `ads[]` | Paid results on the page |
| `people_also_ask[]` | Questions from the "People also ask" box |
| `related_searches[]` | Related queries |
| `local_results[]` | Local pack results for queries with local intent |

## Localization: gl, hl and location

- **`gl`** is the country the search runs for, as a two-letter code such as `us` or `de`. It decides which country's rankings you see.
- **`hl`** is the interface language, such as `en` or `de`. It changes labels and can shift which language versions of pages rank.
- **`location`** narrows the search to a city or region, such as `Chicago,Illinois`. Local packs and many commercial queries differ completely from one city to the next.

In a DIY scraper, Google infers location from the IP, and the `uule` URL parameter can encode a more precise location. Whatever you use, keep the signals aligned: a Chicago IP with `gl=us` and `hl=en`. Mixed signals, such as a German IP with `gl=us`, produce blended results that match no real searcher.

## Pagination, devices and request rates

**Pagination.** Each page is one request: `page=1`, `page=2` and so on in the API, or `start=0`, `start=10` in Google URLs. Decide how deep you really need to track. For [rank tracking](https://proxonym.com/use-cases/serp-tracking), the top 30 positions for 1,000 keywords once a day is 3 pages × 1,000 keywords × 30 days = 90,000 requests a month, which fits the 100,000-request plan.

**Devices.** Mobile and desktop results differ in layout, features and sometimes rankings. Set `device=mobile` or `device=desktop` in the API; in a DIY scraper, use a matching user agent and viewport. Track both if most of your audience searches on phones.

**Rates for DIY.** Rotate IPs per keyword, spread queries evenly over time with random jitter, and avoid bursts at the top of the hour when scheduled jobs tend to start. Detect `/sorry/` redirects, `429` responses and consent interstitials, and treat a rising share of them as a signal to slow down rather than to retry harder. A simple check covers the common cases:

**Python**

```python
def is_blocked(final_url, status, html):
    return (
        status == 429
        or "/sorry/" in final_url
        or "unusual traffic" in html.lower()
    )
```

Count every blocked response per hour and per exit country, and cut the rate when the share climbs. Do not route queries through CAPTCHA-solving services; lower the rate instead. The general techniques are in [web scraping without getting blocked](https://proxonym.com/blog/web-scraping-without-getting-blocked).

**Caching.** Rankings move daily, not by the minute. Store each keyword's results with a date and reuse them for the rest of the day; a dashboard refresh should never trigger a new query.

## Legal considerations

Collecting publicly visible search results is common practice in SEO and market research, but it is not without rules. Google's terms of service restrict automated access, and its robots.txt disallows crawling of search pages. Whether that creates liability depends on your jurisdiction and on how you use the data. Collect only what you need, avoid storing personal data that appears in results, do not republish copyrighted snippets wholesale, and get legal advice for commercial projects. Our [acceptable use policy](https://proxonym.com/acceptable-use) applies to every request, whichever approach you choose.

## Key takeaways

- DIY Google scraping needs residential IPs, a headless browser, a parser and block handling, so budget for ongoing maintenance.
- A SERP API returns parsed JSON at one request per results page, from $0.16 per 1,000 requests.
- Align `gl`, `hl`, location and IP, or you will measure results no real searcher sees.
- Google shows about ten organic results per page, so plan your depth: the top 100 costs ten requests per keyword.
- Throttle, back off instead of solving CAPTCHAs, and respect the legal limits of your use case.

Put it into practice

- [Residential proxies](https://proxonym.com/residential-proxies): Best for scraping well-protected sites. from **$0.74**/GB
- [Unlimited residential](https://proxonym.com/unlimited-residential-proxies): Best for heavy crawls and AI training data. from **$33.25**/day

Step-by-step setup

- [Python Requests proxy setup](https://proxonym.com/integrations/python-requests)
- [Node.js proxy setup](https://proxonym.com/integrations/nodejs)

Try it on your own target

Self-serve plans, crypto accepted, live in minutes. Plans never renew on their own.

## Keep reading

[All articles](https://proxonym.com/blog)

- [Proxies for AI Agents: Reliable Web Access for LLM Apps](https://proxonym.com/blog/proxies-for-ai-agents): AI agents fail when the web blocks them. Which tool fits each job, from search grounding to browsing agents, with Playwright code and the rules that keep agents welcome. (Guides, 5 min read)
- [How to Choose a Residential Proxy Provider: 10 Checks](https://proxonym.com/blog/choose-residential-proxy-provider): Headline prices hide most of the cost. Ten checks that tell residential proxy providers apart, a table of red flags and a simple way to test on your own targets. (Guides, 5 min read)
- [Web Scraping Without Getting Blocked: 12 Practical Techniques](https://proxonym.com/blog/web-scraping-without-getting-blocked): Twelve techniques that keep scrapers running: respect the rules, pace requests, rotate IPs sensibly, stay consistent, back off on errors, cache and monitor success rates. (Guides, 5 min read)
