A frozen node is twice as expensive as an absent one, and marks its peer's completed work as failed

Two observations from the same failure mode, measured on a two-node cluster (Ubuntu + RTX 3060, macOS, v0.1.1-463). Both matter for a cluster that includes a laptop, since a laptop that sleeps mid-request is exactly this shape: the socket is accepted, nothing answers.

Setup

24 concurrent chat/completions, max_tokens: 200, google/gemma-4-12b resident on both nodes.

Every request returned 200 in all three. Reliability is not the problem — the cost is.

A frozen peer is 2.1× worse than not having it at all, and 4.1× worse than a healthy cluster. When I instead killed the peer's processes outright, the same load finished in 28.3 s with no failures: a closed socket gives an immediate RST and the proxy moves on in milliseconds. It is specifically the accepted-but-silent socket that hurts.

For context on the eviction path: after the peer was killed, discovery had dropped it from the node list within 75 s — I sampled too coarsely to pin it more precisely. During that window each request pays its own connection timeout.

The second half: false failures

When the frozen peer resumed, its job history marked several of the other node's workloads as

Failed — node serveri stopped reporting this workload

Those workloads had all completed successfully; the requesting node returned 200 for all 24. The peer's view had simply gone stale while it was stopped, and on resume it reaped them as failed. The timestamps match the resume to the second.

That leaves the job history — the only place a user sees what the cluster did — showing failures that did not happen, precisely in the situation where something unusual was going on and they would be looking at it.

Not filed as a defect

Neither is a crash and both may be deliberate. But if a laptop is a first-class node, asleep with the socket accepted and nothing answering is its most likely state, not an edge case. Two directions that would help, if either is welcome:

• a connection-level timeout distinctly shorter than the request timeout, so an unresponsive candidate is abandoned quickly while a slow-but-live generation is not
• reaping a peer's workloads as unknown rather than failed when the gap was on our side

Happy to open a PR for either if you have a preferred shape.