Call the cancel endpoint eagerly when the editor abandons a stream

Release blocker — the client half of an InferenceInfra fix. Depends on the sidecar exposing POST /v1/requests/{id}/cancel and the gateway forwarding x-request-id (RunanywhereAI/InferenceInfra#440).

Problem

When the editor abandons a stream (Esc in Claude Code, app quit), the shim today drops the upstream request only implicitly: the upstream content receiver in src/anthropic/messages.cpp returns false when sink.write to the editor fails (~L238), which makes cpp-httplib close the socket. Two gaps:

• It fires only when the next upstream chunk arrives. During a long prefill (a 100k-token coding prompt is exactly when a user gives up) nothing arrives, so nothing fires.
• It is only a TCP close, and that is the signal the managed edge loses: measured through the public hostname, the engine keeps decoding 22–41 s after the client is gone, and the customer is billed for it.

Fix

1. Capture the x-request-id from the upstream response headers as soon as they arrive (before the first body chunk).
2. Detect the editor's close eagerly — a failed write, the connection's close callback, or the process exiting — rather than waiting for the next chunk.
3. On close, POST {base_url}/requests/{id}/cancel with the session's bearer, fire-and-forget with a short timeout, in addition to dropping the upstream socket. Log it to shim.log, never to the editor's terminal.
4. On normal completion, do nothing.

Done when

Abandoning a stream from Claude Code through wally stops the engine within 2 s (the InferenceInfra issue's acceptance), including when abandoned during prefill.

Related: the keep-alive issue (same file — coordinate so the pooled client owns the header capture).

Implementation handoff — 2026-09-13

Status: active teammate work; no dedicated #81 PR published at this check. Aman owns the reviewed cancellation implementation; Siddhesh coordinates Wally merging. Sanchit revalidated the issue and avoided a competing fix. This issue remains open, and cancellation is not claimed implemented or live-validated by this handoff.

Current merge sequence for the overlapping translator code: #82 → #103 → #104, then rebase Aman's #81 branch onto the resulting main (or the agreed stack head before those merges). #103 currently targets issue-80-shim-keepalive; #104 targets codex/issue83-stream-overload. Retarget dependent PRs to main after their prerequisites land. Re-run abandonment tests with the overload and incomplete-stream changes present.

Server prerequisite: InferenceInfra #475, targeting development, must be deployed to the relevant control plane and nodes before end-to-end cancellation acceptance.

Acceptance clarification: the reviewed pinned LiteLLM path delays the request ID until the first token. Consequently, the client change alone cannot meet the original two-second cancellation target during a longer prefill; the gateway early-ID requirement remains tracked with InferenceInfra #440. Validate cancellation after ID arrival, no replay after abandonment, bounded shutdown, both translators, and final engine-stop/billing behavior. Preserve the original prefill requirement as outstanding.

Detailed final handoff: https://github.com/RunanywhereAI/wally/issues/81#issuecomment-5651887056