gosom / gosom/google-maps-scraper
Option To Not Retry Failed Jobs?
- Dominant language
- Go
- Stars
- 5.9k
- Forks
- 924
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 3
Description
Hi, I'm making this automation program that uses the scraper. Speed is quite important to me, but when the scraper has a failed job, it keeps retrying very slowly. I get the following output, for example:
```
{"level":"info","component":"scrapemate","numOfJobsCompleted":98,"numOfJobsFailed":6,"lastActivityAt":"2025-06-30T09:36:20.868298826Z","speed":"98.00 jobs/min","time":"2025-06-30T09:36:26.66053663Z","message":"scrapemate stats"}
{"level":"info","component":"scrapemate","numOfJobsCompleted":98,"numOfJobsFailed":6,"lastActivityAt":"2025-06-30T09:36:20.868298826Z","speed":"49.00 jobs/min","time":"2025-06-30T09:37:26.660376592Z","message":"scrapemate stats"}
```
The "jobs/min" keep going down, and the program usually hangs like this for at least a couple minutes. I've fixed this by lowering the -exit-on-inactivity flag, but I'd rather it not wait until that duration is set. I think it'd be a good feature to have a flag like "-dont-retry-failed" or something like that, that just tells the program to exit, without retrying the failed jobs.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the handling of the -exit-on-inactivity flag and the code that retries failed jobs. Define the new flag's behavior so failed jobs are not retried and the program exits instead; verify the scraper no longer waits on those retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100