Global service updates get stuck if a node can't run the task
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
The approach for updates makes sense for replicated services. When parallelism is set to 1, we update one slot at a time, and don't move on to the next slot until the update suceeds. There would be no reason to move on, because all slots are identical.
It's not quite the same with global services. Each task is bound to one node, so if something is wrong with a particular node, or it doesn't have enough available resources for the task to be moved to Assigned, the update will wait forever at that point. Should we have a timeout on waiting for the new task to start up, after which we either pause or continue depending on FailureAction?
cc @aluzzardi @dongluochen
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the global service update path and how it waits for a replacement task to reach Assigned or running. Compare that behavior with replicated-service updates and inspect how FailureAction is handled. Done means defining and implementing a timeout policy for unavailable nodes, including whether the update pauses or continues, with tests for both outcomes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100