gosom / gosom/google-maps-scraper
Preserve Maps data IDs when detail URLs contain a dot-dot place segment
- Dominant language
- Go
- Stars
- 5.9k
- Forks
- 924
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 3
Description
## Problem
A Maps feed can emit a direct-place URL whose path contains the exact segment form:
```text
https://www.google.com/maps/place/../data=
```
Passing that URL unchanged to the detail job allows standard path normalization to collapse `place/..`. The parent reports the place as discovered, but the detail fetch can then fail and the serialized result is lost. We reproduced a stable one-detail deficit across three attempts while the parent discovery count remained higher.
## Proposed behavior
Before constructing the detail job:
1. Parse and require the canonical HTTPS `www.google.com` authority, with no userinfo or port.
2. Match exactly one `/maps/place/../data=` marker in the path and exactly one strong Maps data ID in its payload.
3. Replace only the `..` path segment with `_`, preserving the `data=` payload byte-for-byte.
4. Optionally fall back to `https://www.google.com/maps?cid=` derived from the second half of the strong data ID.
5. Reject any ambiguous form rather than guessing.
6. Emit structured detail events carrying `parent_id`, job type, URL SHA-256, and outcome so losses are attributable to the parent.
A reviewed implementation and regressions are available in GinkgoBil/maps-crawler#141. The patch is based on upstream v1.16.0 and its full `go test ./...` suite passes. We are keeping the fork identity explicit (`v1.16.0+maps-crawler.1`) and will not present it as vanilla upstream.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the detail-job construction and run `go test ./...`; the issue provides no repository file or test path. Use GinkgoBil/maps-crawler#141 as the reviewed implementation reference, with regressions covering URL validation and rewriting, optional CID fallback, ambiguous-form rejection, and structured attribution events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100