Failure-aware scheduling (don't reassign task to faulty node endlessly)
Open
Nobody has claimed this yet.
area/orchestration
priority/P2
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Right now the RestartSupervisor will restart a failed task after (at least) RestartDelay.
We should instead use an exponential backoff, from 0 (immediate restart) to some sensitive value (few hours).
It means that:
- An occasional crash will recover immediately rather than waiting restart delay (5 seconds right now)
- A permanent or semi-permanent "crash" (e.g. bad image) will eventually retry every few hours rather than every few seconds
/cc @aaronlehmann @dongluochen
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate RestartSupervisor and trace how it currently applies RestartDelay when restarting failed tasks. Define completion as immediate recovery for occasional crashes and progressively longer retries, reaching an interval of a few hours for persistent failures, while preventing endless rapid reassignment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100