gosom / gosom/google-maps-scraper
Supervisor stops dispatching under sustained FIFO load: jobs stay 'pending' forever, only a restart recovers
- Dominant language
- Go
- Stars
- 5.9k
- Forks
- 924
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 3
Description
## Summary
Under sustained FIFO load, the scraping supervisor quietly dies: every remaining job stays `pending` forever, the API keeps working, and **nothing short of a full process restart recovers it**. Related to #279 and #283 (same symptom family, but our repro run and the extra observations below should help narrow it down).
## Environment
- Image: `gosom/google-maps-scraper:latest` (digest `sha256:9442f4332ee0…`, pulled 2026-08-23), REST API mode (`/api/v1/jobs`).
- Workload: a pipeline that gradually POSTs thousands of small **targeted** jobs: `keywords: [""]`, `lat`/`lon`, `radius: 150`, `fast_mode: false`, `depth: 1`, `max_time: 60–300s`, `email: false`, `zoom: 15`. Jobs are consumed by 2–5 concurrent workers.
## Observed behavior
1. After ~120–250 `ok` jobs over 1.5–2 h, the supervisor stops dispatching. All queued jobs stay `pending` forever; none ever transitions to `working` again. **No error is logged** — the last line is exactly:
```
{"level":"info","component":"scrapemate","time":"2026-09-03T10:08:04.24714673Z","message":"scrapemate exited"}
```
2. In the same session a job was stuck in `working` for **20+ minutes** although its `max_time` was 60 s. `max_time` appears **not to be enforced** once the browser hangs (chromium deadlock): status stays `working` indefinitely and the whole FIFO behind it is blocked.
3. `DELETE /api/v1/jobs/:id` removes the job record (HTTP 200) but does **not** unblock the supervisor: the next `pending` job still never starts. Freezing/unfreezing or killing the browser process does not help either.
4. The HTTP API keeps responding normally throughout (create/status/delete all work) — only the dispatch loop is dead.
## Recovery
Only a full container/process restart fixes it. After restart, queue processing resumes normally (in our system the workers detect the dropped jobs and re-submit them).
## Repro hints
Sustained FIFO load of small jobs over ~1–2 h (same as #283's report) — may require a few thousand submissions before the supervisor dies. Happy to attach the docker-compose + a small submitter script if that helps.
## Impact
Bad for automation: a silent dead pipeline that can only be recovered by an OOM-kill style restart (liveness probes/restart policies in orchestrators do not trigger on a still-alive but dead-loop process).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the /api/v1/jobs entry points and trace the supervisor dispatch loop around the reported "scrapemate exited" event. Reproduce sustained FIFO submissions while observing pending and working jobs, then inspect timeout and cancellation handling. Done means the supervisor continues dispatching, max_time is enforced for hung jobs, and a regression test covers recovery without a process restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100