grpc-proxy: stateful session rejoin cannot detect a session whose worker is gone

Summary

When a client reconnects to an existing stateful gRPC session, grpc-proxy hands the session back to its worker by publishing a reconnect message on stateful_session.reconnect.<requestId>. This is a plain core NATS publish, which succeeds whether or not anything is subscribed.

If the worker that owned the session is no longer there, the message is discarded by the server and the publish still returns success. The proxy reports a healthy rejoin, waits for a worker that cannot arrive, and the client is left holding a session cookie it presents again on every retry. No error is surfaced on any path, so nothing recovers.

Impact

Sessions in this state produce sustained client-visible errors and do not self-heal. The only reliable remedy today is restarting or rolling over the function, which is operator action for what should be an automatic recovery.

Why it does not recover on its own

The recovery mechanism already exists. StreamDirector.ServeHTTP clears the client's request id cookie when a request fails with ErrSessionNotFound, and a client that drops the cookie opens a fresh session on its next request. That path is never reached, because the rejoin has no way to learn that the session is dead and so never produces ErrSessionNotFound.

Two smaller gaps contribute:

• The worker's reconnect listener does not acknowledge receipt, so subscription interest is the only available liveness signal.
• The worker subscribes to the reconnect subject only after its first CONNECT completes, leaving a window during session establishment where a live session has nothing listening.

Precedent

The stateless polling path already does this correctly. polling_request uses request/reply and maps a no-responders answer onto "no worker picked this up", and the caller turns that into a clear client-visible error. The worker's polling listener acknowledges receipt. The stateful reconnect path is the same shape minus the acknowledgement and the delivery check.

Proposed fix

Make the rejoin a request rather than a publish, treat a confirmed no-responders answer as ErrSessionNotFound, acknowledge reconnects on the worker side, and subscribe before the first CONNECT.

Out of scope

This addresses sessions that cannot be handed back to a worker. It does not address worker CONNECT tokens expiring while work waits for a concurrency slot, which is a separate failure mode.

Start by tracing StreamDirector.ServeHTTP and the stateful reconnect listener, then compare them with the polling_request request/reply path. Check the worker's reconnect subscription timing around the first CONNECT. Done means dead-session rejoins produce ErrSessionNotFound and clear the cookie, while live workers acknowledge reconnects reliably.