grpc-proxy: queued stateful work requests outlive the tokens needed to authenticate them

Start in the grpc-proxy graceful-shutdown path and compare its session handling with the invocation service's existing cancel_request purge behavior. Trace the subjects for tokens minted by the departing pod through the JetStream work queue; done means purging only still-queued requests while leaving sessions that already connected unaffected.

Summary

grpc-proxy mints a worker CONNECT token per stateful invocation and keeps it in a per-pod in-memory cache. The corresponding work request is durable: it sits in the JetStream work queue until a worker has a free concurrency slot to pull it.

When a proxy pod goes away, every token it minted goes with it. Any work request it issued that has not yet been pulled is now guaranteed to fail: a worker eventually pulls it, takes a concurrency slot, tries to CONNECT, is rejected with 403, and gives the slot back having done nothing useful.

Nothing removes those requests. On a saturated function the queue is deep enough that this repeats for as long as the backlog takes to drain, while clients retry and refill it.

Impact

After a proxy restart, a busy function can spend an extended period at or near zero goodput, working through a backlog in which every request is dead on arrival. Operators currently break the cycle by dropping demand, for example by scaling the function down and back up.

Proposed fix

On shutdown, purge the queued work requests for sessions this pod issued a token for that never came back to CONNECT. Those are precisely the requests that can no longer succeed.

The invocation service already does this for a cancelled request: cancel_request purges the request's subject from the same stream.

Important scoping constraint

Only requests still waiting for a worker may be purged.

A session that already has a worker attached is not pinned to the proxy pod that started it. On reconnect the proxy builds the connection config from the answering pod's own address and mints a fresh token, so the worker reattaches to a different pod and the session survives. Purging state for those sessions would sever sessions that were going to survive the restart, turning a clean rolling update into a mass session reset.

Purging by subject only removes messages the stream still holds, so an established session, whose message has already been delivered, is naturally unaffected.

Known limitations

This runs during graceful shutdown, so it covers a rolling update but not a hard kill, node loss, or OOM. It also depends on this service having purge rights on the work queue, which is granted outside this repository.