gosom / gosom/google-maps-scraper
-extra-reviews returns ~10 reviews: RPC listugcposts returns 403 (v1.17.4)
- Dominant language
- Go
- Stars
- 5.9k
- Forks
- 924
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 3
Description
### What happens
With `-extra-reviews`, `user_reviews_extended` returns 2 items instead of ~300. The logs show the RPC endpoint being rejected before the DOM fallback kicks in:
```
Browser-based RPC fetch failed: fetch error: HTTP 403, trying HTTP
RPC fetch failed, will try DOM extraction: .../maps/rpc/listugcposts...: unexpected status code: 403
Attempting DOM-based review extraction
Review count stuck at 10, stopping scroll
DOM extraction completed: 10 reviews found
DOM reviews: dropped 8 of 10 already present in user_reviews
```
Result per place: `user_reviews` = 8, `user_reviews_extended` = 2 — on places whose `review_count` is 435, 1034 and 2076.
### What I ruled out
- **Not the version** — `docker pull` confirmed `v1.17.4-beca11f` is current.
- **Not the field** — `user_reviews_extended` does exist, it just contains 2 entries.
- **Not the IP** — same 403 from a home IP and from 4 different US residential proxy exits (RCN New York, Spectrum Ohio, Ashburn VA). Plain Google Maps pages return HTTP 200 through those same proxies, so routing is fine.
The 403 seems to be about the shape of the RPC request rather than where it comes from.
### Environment
Docker `gosom/google-maps-scraper:latest` (`v1.17.4-beca11f`), flags `-extra-reviews -json -lang en -depth 1 -c 1`, queries in Houston TX.
### Side note that may save someone a day
Proxies that intercept TLS (Bright Data's residential zone) break the Playwright browser and surface as `Cannot read properties of null (reading 'scrollHeight')` — the page never loads at all, so it looks like a scraping bug rather than a TLS one. Proxies that only tunnel work fine.
Thanks for the tool — the listing and email harvesting side works great.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the `-extra-reviews` path with the listed Docker flags and inspect the code handling the `listugcposts` RPC before its DOM fallback. Compare the RPC request shape with the 403 response and verify that review extraction is no longer capped at 10 while preserving the fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100