Anthropic shim opens a new TLS connection for every request (~541 ms per turn)

Release blocker. ~541 ms is added to every request an agent makes through wally, before a single byte of the request is sent.

Problem

The Anthropic→OpenAI shim builds a fresh httplib::Client for every request to the hosted endpoint: inside the streaming sink in src/anthropic/messages.cpp (HandleStreaming, httplib::Client client(*origin) ~L183) and again in HandleNonStreaming (~L119). Each one opens a new TCP connection and completes a new TLS handshake, streams one response, and is destroyed.

Against the regional load balancer that handshake was measured at ~541 ms (InferenceInfra .claude/rules/terraform-safety.md, 2026-09-06; InferenceInfra #418 measured ~556 ms per new connection from Delhi, and nothing on reused connections). Claude Code makes several requests per user turn once ANTHROPIC_BASE_URL points at the shim — the main completion plus the small side calls — and every one pays it. Claude Code → shim is loopback and unaffected; the shim → inference.runanywhere.ai leg is where the cost lands.

Why it blocks release

It is the largest piece of time-to-first-token a customer feels that has nothing to do with the GPU, on every turn. It also makes the regional-TLS question (InferenceInfra #299) look worse than it is: once the connection is reused, that handshake is paid once per session instead of once per request.

Fix

1. Keep long-lived httplib::Clients on the shim's Runtime, set_keep_alive(true). An httplib::Client serialises requests on its one socket, so use a small pool — one per concurrent stream — and reuse across turns.
2. Reconnect lazily: if a request fails on a stale socket (the ALB keeps idle client connections ~10 minutes), retry once on a fresh one. Never retry a request that has started streaming.
3. Enable TLS session resumption on the client so a reconnect is one round-trip.
4. Same pattern in src/harness/codex.cpp / opencode.cpp only if they own an HTTP client of their own; if they hand the tool a base URL and the tool's client talks directly, they are unaffected.

Done when

A packet capture of a second request in the same session shows no TLS handshake, and the per-turn TTFT in a real agent session drops by roughly the handshake time.

Related: the cancel-on-close issue (same file), InferenceInfra #299.

Read src/anthropic/messages.cpp at HandleStreaming and HandleNonStreaming, then trace how the shim Runtime owns outbound clients. Check the related client paths in src/harness/codex.cpp and opencode.cpp only if they create HTTP clients. Done means a second request reuses the connection without a TLS handshake, stale sockets reconnect safely, and real-agent per-turn TTFT drops by roughly the handshake time.