TraceMachina / TraceMachina/nativelink
Give a grace period for a dropped execution request
@MarcusSorealheis is already working on this.
Since Aug 31, 2025.
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 244
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 54
Description
Currently, if an action is queued and not being actively monitored by a client (the client drops the Execute or WaitExecution request) then the action is removed from the queue. However, sometimes a network blip or GKE re-allocation causes the scheduler to drop the connection to the client and therefore cancel the jobs that are still very much required from the queue.
Instead, we should give a grace period of a few seconds for the client to re-connect before removing the action from the queue. Note that this is slightly complicated by the re-connection potentially happening on a different node in a distributed system.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.