Provider stream disconnects fail the node outright with no reconnect or resume

Summary

A provider SSE stream drop (stream disconnected before completion) fails the node outright. There is no transport-level reconnect or resume, so a network event that has nothing to do with the model's work discards the whole node — and with retries: 0 in the default profile, discards it permanently.

Evidence

From an 18-hour local default-profile run, five NodeFailed events of this class:

+09:07:25  NodeFailed  ✗ node:<dynamic-goal-a> (attempt 2): stream disconnected before completion
+09:14:11  NodeFailed  ✗ node:<dynamic-goal-b> (attempt 2): stream disconnected before completion
+10:58:09  NodeFailed  ✗ node:<dynamic-goal-c> (attempt 1): stream disconnected before completion
+11:09:07  NodeFailed  ✗ node:<dynamic-goal-d> (attempt 1): stream disconnected before completion

All on long-running goal nodes under a max-reasoning routed model.

Why this class of failure is structural, not incidental

High reasoning effort on a routed provider means long stretches with no tokens on the wire. Gateways and intermediaries time idle SSE connections out. So the exact configuration the product recommends for its most valuable nodes is the configuration most likely to produce idle-stream drops — the failure rate rises with reasoning effort and node duration, which is precisely backwards from what an operator wants.

A dropped stream is also the cheapest possible failure to recover from in principle: the request is usually still being served, and the work already done is not invalid. Treating it as a terminal node failure throws away up to two hours of paid model output because a socket closed.

Proposal

• Reconnect or resume the stream on disconnect rather than failing the node, with a bounded number of reconnects.
• Where the harness supports resuming a session by id, prefer resuming over re-running: agent trace events already record a resume thread id per node, so the hook exists.
• Emit a keepalive or poll during long silent reasoning phases so idle intermediaries do not close the connection in the first place.
• Failing that, classify stream disconnects as retryable transport errors that consume a retry budget rather than terminating the node — which also depends on the default profile actually having a retry budget (filed separately).

Acceptance criteria

• A stream disconnect during a long reasoning phase does not fail the node while the underlying request is still viable.
• Reconnect attempts are bounded and logged with a distinct diagnostic, so genuine provider outages still fail fast.
• A regression test simulates a mid-stream disconnect and asserts the node completes.

Locate the provider SSE stream handling and node execution path, then inspect how the agent trace's resume thread id is used. Review the existing retry budget and diagnostics before choosing a bounded recovery approach. Done means a mid-stream disconnect regression test shows the node completes, with bounded, logged reconnect attempts and genuine outages still failing.