Monitoring crashed workers and rescuing stuck jobs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.7k
- Forks
- 179
- Avg merge
- 15h 43m
- Merged PRs (30d)
- 13
Description
Hello there!
After reading about leader election, updating timestamps, my initial thought was that rescuer would also monitor all workers that have running jobs attached to them.
I assumed that all workers would update some kind of a healthcheck timestamp every 5 seconds or so. And if some worker hasn't updated their timestamp in a reasonable time period, all its currently running jobs should be rescued.
However, I see that jobs are only rescued based on a global rescue timeout.
What do you think about this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the rescuer, worker, leader-election, and global rescue-timeout behavior described in the issue. Compare the proposed worker healthcheck timestamps with the current rescue model and identify failure cases and design constraints. Done means the project has an agreed approach and clear behavior for detecting failed workers and rescuing their jobs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100