gosom / gosom/google-maps-scraper

Option To Not Retry Failed Jobs?

Open
#158 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
5.9k
Forks
924
Avg merge
5d 13h
Merged PRs (30d)
3

Description

Hi, I'm making this automation program that uses the scraper. Speed is quite important to me, but when the scraper has a failed job, it keeps retrying very slowly. I get the following output, for example:

```
{"level":"info","component":"scrapemate","numOfJobsCompleted":98,"numOfJobsFailed":6,"lastActivityAt":"2025-06-30T09:36:20.868298826Z","speed":"98.00 jobs/min","time":"2025-06-30T09:36:26.66053663Z","message":"scrapemate stats"}
{"level":"info","component":"scrapemate","numOfJobsCompleted":98,"numOfJobsFailed":6,"lastActivityAt":"2025-06-30T09:36:20.868298826Z","speed":"49.00 jobs/min","time":"2025-06-30T09:37:26.660376592Z","message":"scrapemate stats"}
```

The "jobs/min" keep going down, and the program usually hangs like this for at least a couple minutes. I've fixed this by lowering the -exit-on-inactivity flag, but I'd rather it not wait until that duration is set. I think it'd be a good feature to have a flag like "-dont-retry-failed" or something like that, that just tells the program to exit, without retrying the failed jobs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the handling of the -exit-on-inactivity flag and the code that retries failed jobs. Define the new flag's behavior so failed jobs are not retried and the program exits instead; verify the scraper no longer waits on those retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.