TraceMachina / TraceMachina/nativelink
Worker occasionally dies with "UpdateForWorker stream closed early"
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 244
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 54
Description
2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
at nativelink-worker/src/running_actions_manager.rs:1415
in nativelink_worker::local_worker::worker_start_action
in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
at nativelink-worker/src/running_actions_manager.rs:1415
in nativelink_worker::local_worker::worker_start_action
in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
at nativelink-worker/src/local_worker.rs:701
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with nativelink-worker/src/local_worker.rs:701 and running_actions_manager.rs:1415, then trace the UpdateForWorker stream handling up to src/bin/nativelink.rs:702. Reproduce the worker disconnect using the reported logs and determine why actions in transit do not reach zero before the stream closes; done means the failure is understood and the worker no longer exits unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100