gosom / gosom/google-maps-scraper

-extra-reviews returns ~10 reviews: RPC listugcposts returns 403 (v1.17.4)

Open
#327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.9k
Forks
924
Avg merge
5d 13h
Merged PRs (30d)
3

Description

### What happens

With `-extra-reviews`, `user_reviews_extended` returns 2 items instead of ~300. The logs show the RPC endpoint being rejected before the DOM fallback kicks in:

```
Browser-based RPC fetch failed: fetch error: HTTP 403, trying HTTP
RPC fetch failed, will try DOM extraction: .../maps/rpc/listugcposts...: unexpected status code: 403
Attempting DOM-based review extraction
Review count stuck at 10, stopping scroll
DOM extraction completed: 10 reviews found
DOM reviews: dropped 8 of 10 already present in user_reviews
```

Result per place: `user_reviews` = 8, `user_reviews_extended` = 2 — on places whose `review_count` is 435, 1034 and 2076.

### What I ruled out

- **Not the version** — `docker pull` confirmed `v1.17.4-beca11f` is current.
- **Not the field** — `user_reviews_extended` does exist, it just contains 2 entries.
- **Not the IP** — same 403 from a home IP and from 4 different US residential proxy exits (RCN New York, Spectrum Ohio, Ashburn VA). Plain Google Maps pages return HTTP 200 through those same proxies, so routing is fine.

The 403 seems to be about the shape of the RPC request rather than where it comes from.

### Environment

Docker `gosom/google-maps-scraper:latest` (`v1.17.4-beca11f`), flags `-extra-reviews -json -lang en -depth 1 -c 1`, queries in Houston TX.

### Side note that may save someone a day

Proxies that intercept TLS (Bright Data's residential zone) break the Playwright browser and surface as `Cannot read properties of null (reading 'scrollHeight')` — the page never loads at all, so it looks like a scraping bug rather than a TLS one. Proxies that only tunnel work fine.

Thanks for the tool — the listing and email harvesting side works great.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the `-extra-reviews` path with the listed Docker flags and inspect the code handling the `listugcposts` RPC before its DOM fallback. Compare the RPC request shape with the 403 response and verify that review extraction is no longer capped at 10 while preserving the fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.