TraceMachina / TraceMachina/nativelink

thread 'tokio-runtime-worker' panicked at nativelink-service/src/bytestream_server.rs:249:41

Open
#1,889 7 comments 0 reactions 1 assignee View on GitHub

@MarcusSorealheis is already working on this.

Since Aug 15, 2025.

Dominant language
Rust
Stars
1.6k
Forks
244
Avg merge
1d 16h
Merged PRs (30d)
54

Description

I'm getting many warnings in my cas service logs:

thread 'tokio-runtime-worker' panicked at nativelink-service/src/bytestream_server.rs:249:41:
`async fn` resumed after completion
  2025-08-07T15:13:49.779565Z  WARN nativelink_util::buf_channel: Stream already closed when eof already was sent. This is often ok for retry was triggered, but should not happen on happy path.
    at nativelink-util/src/buf_channel.rs:158
    in nativelink_service::bytestream_server::inner_write with digest: DigestInfo("e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222-21875"), stream: WriteRequestStreamWrapper { resource_info: ResourceInfo { instance_name: "", uuid: Some("a0fef723-55a2-49d3-a032-70cba80d4512"), compressor: None, digest_function: Some("sha256"), hash: "e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222", size: "21875", expected_size: 21875, optional_metadata: None }, bytes_received: 0, first_msg: Some(WriteRequest { resource_name: "/uploads/a0fef723-55a2-49d3-a032-70cba80d4512/blobs/sha256/e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222/21875", write_offset: 16384, finish_write: false, data: b"<redacted>" }), write_finished: false }, stream.first_msg: "<redacted>"
    in nativelink_service::bytestream_server::bytestream_write
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 10.55.138.219:40498, socket_addr: 0.0.0.0:50051

Nativelink version: 0.6.0

CAS config:

{
  "stores": {
    "cas_FAST_SLOW_STORE": {
      "verify": {
        "backend": {
          "fast_slow": {
            "fast": {
              "size_partitioning": {
                "size": 64000,
                "upper_store": {
                  "noop": {}
                },
                "lower_store": {
                  "memory": {
                    "eviction_policy": {
                      "max_bytes": "32Gb",
                      "max_count": 100000000
                    }
                  }
                }
              }
            },
            "slow": {
              "filesystem": {
                "content_path": "/opt/nativelink/data/cas",
                "temp_path": "/opt/nativelink/data/tmp",
                "eviction_policy": {
                  "max_bytes": "400Gb"
                }
              }
            }
          }
        }
      },
      "verify_size": true,
      "hash_verification_function": "sha256"
    },
    "cas_STORE": {
      "existence_cache": {
        "backend": {
          "ref_store": {
            "name": "cas_FAST_SLOW_STORE"
          }
        },
        "eviction_policy": {
          "max_count": 1000000000,
          "max_seconds": 180000
        }
      }
    }
  },
  "servers": [
    {
      "listener": {
        "http": {
          "socket_address": "0.0.0.0:50051",
          "compression": {
            "send_compression_algorithm": "gzip",
            "accepted_compression_algorithms": [
              "gzip"
            ]
          },
          "advanced_http": {
            "experimental_http2_keep_alive_timeout": 1200
          },
          "tls": {
            "cert_file": "/opt/nativelink/cas.crt",
            "key_file": "/opt/nativelink/cas.key"
          }
        }
      },
      "services": {
        "cas": {
          "": {
            "cas_store": "cas_STORE"
        }
        },
       "capabilities": {},
        "bytestream": {
          "cas_stores": {
            "": "cas_STORE"
          }
        }
      }
    },
    {
      "listener": {
        "http": {
          "socket_address": "0.0.0.0:50061"
        }
      },
      "services": {
        "health": {}
      }
    }
  ]
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.