gosom / gosom/google-maps-scraper

Supervisor stops dispatching under sustained FIFO load: jobs stay 'pending' forever, only a restart recovers

Open
#328 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.9k
Forks
924
Avg merge
5d 13h
Merged PRs (30d)
3

Description

## Summary
Under sustained FIFO load, the scraping supervisor quietly dies: every remaining job stays `pending` forever, the API keeps working, and **nothing short of a full process restart recovers it**. Related to #279 and #283 (same symptom family, but our repro run and the extra observations below should help narrow it down).

## Environment
- Image: `gosom/google-maps-scraper:latest` (digest `sha256:9442f4332ee0…`, pulled 2026-08-23), REST API mode (`/api/v1/jobs`).
- Workload: a pipeline that gradually POSTs thousands of small **targeted** jobs: `keywords: [""]`, `lat`/`lon`, `radius: 150`, `fast_mode: false`, `depth: 1`, `max_time: 60–300s`, `email: false`, `zoom: 15`. Jobs are consumed by 2–5 concurrent workers.

## Observed behavior
1. After ~120–250 `ok` jobs over 1.5–2 h, the supervisor stops dispatching. All queued jobs stay `pending` forever; none ever transitions to `working` again. **No error is logged** — the last line is exactly:
```
{"level":"info","component":"scrapemate","time":"2026-09-03T10:08:04.24714673Z","message":"scrapemate exited"}
```
2. In the same session a job was stuck in `working` for **20+ minutes** although its `max_time` was 60 s. `max_time` appears **not to be enforced** once the browser hangs (chromium deadlock): status stays `working` indefinitely and the whole FIFO behind it is blocked.
3. `DELETE /api/v1/jobs/:id` removes the job record (HTTP 200) but does **not** unblock the supervisor: the next `pending` job still never starts. Freezing/unfreezing or killing the browser process does not help either.
4. The HTTP API keeps responding normally throughout (create/status/delete all work) — only the dispatch loop is dead.

## Recovery
Only a full container/process restart fixes it. After restart, queue processing resumes normally (in our system the workers detect the dropped jobs and re-submit them).

## Repro hints
Sustained FIFO load of small jobs over ~1–2 h (same as #283's report) — may require a few thousand submissions before the supervisor dies. Happy to attach the docker-compose + a small submitter script if that helps.

## Impact
Bad for automation: a silent dead pipeline that can only be recovered by an OOM-kill style restart (liveness probes/restart policies in orchestrators do not trigger on a still-alive but dead-loop process).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the /api/v1/jobs entry points and trace the supervisor dispatch loop around the reported "scrapemate exited" event. Reproduce sustained FIFO submissions while observing pending and working jobs, then inspect timeout and cancellation handling. Done means the supervisor continues dispatching, max_time is enforced for hung jobs, and a regression test covers recovery without a process restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.