TraceMachina / TraceMachina/nativelink

Worker occasionally dies with "UpdateForWorker stream closed early"

Open
#2,115 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.6k
Forks
244
Avg merge
1d 16h
Merged PRs (30d)
54

Description

  2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nativelink-worker/src/local_worker.rs:701 and running_actions_manager.rs:1415, then trace the UpdateForWorker stream handling up to src/bin/nativelink.rs:702. Reproduce the worker disconnect using the reported logs and determine why actions in transit do not reach zero before the stream closes; done means the failure is understood and the worker no longer exits unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.