pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

[grpc-java-pickfirst-wedge.tar.gz](https://github.com/user-attachments/files/30553073/grpc-java-pickfirst-wedge.tar.gz)

### What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

### What is your environment?

Linux, Java 17 (also observed under Java 11 in production).
Load balancing policy: `pick_first` (single-address target).
Active LB implementation: legacy `io.grpc.internal.PickFirstLoadBalancer` (its
`RequestConnectionPicker` appears in the channel logs; the new
`PickFirstLeafLoadBalancer` was not enabled — see "Additional data point" below).

### What did you expect to see?

After a transient network failure heals, a `pick_first` channel either recovers or
transitions out of READY so that the LB policy or the application (via
`notifyWhenStateChanged`) can react.

### What did you see instead?

The channel is left **permanently reporting READY** while every RPC — including
`withWaitForReady()` RPCs — buffers forever and fails only by its own deadline:

```
DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]
```

No channel state transition ever fires again, so nothing can observe that the channel
is broken. `ManagedChannel.enterIdle()` reliably recovers it.

Mechanism, from FINEST `ChannelLogger` logs (attached):

1. The transport to the current address dies (TLS handshakes reset — in production,
a rolling container upgrade left the port published while the backend reset
handshakes for ~10 seconds).
2. gRPC's reconnect logic calls `NameResolver.refresh()` repeatedly. The resolver
answers asynchronously with a mix of `onError` and `onResult` deliveries (it is
backed by a discovery service that sits behind the same failing network, so
deliveries arrive out of order relative to subchannel state changes).
3. Each `onResult` causes `PickFirstLoadBalancer` to create a replacement subchannel
and schedule delayed shutdown (~5s) of the previous one.
4. Within the delayed-shutdown window, the old subchannel completes its connection
against the recovering backend and reaches READY; the LB publishes a READY picker
referencing it.
5. The delayed shutdown then fires (`SHUTDOWN with UNAVAILABLE(Subchannel shutdown
invoked)`), terminating the transport — but the published READY picker is never
replaced. The channel stays READY forever with a picker whose subchannel has no
transport.

### Steps to reproduce the bug

Self-contained repro attached: one Java file (`BurstRepro.java`, only grpc-java +
netty dependencies, in-process backend server), a small Python TCP proxy that models
the network failure (`reset` mode: accept + immediately close, emulating TLS resets),
and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to `reset`
for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy,
then probe with a `withWaitForReady()` RPC with a deadline.

Two wedge variants observed:

- `logs/run2-READY-signature/` — channel READY, wait-for-ready probe black-holes to
`DEADLINE_EXCEEDED`; `enterIdle()` immediately recovers it. This matches our
production incident signature.
- `logs/run1-connecting-wedge/` — channel wedged in CONNECTING with no reconnect
attempt scheduled.

### Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with `GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true`
(confirmed active: `PickFirstLeafLoadBalancer` in the logs), the permanent black-hole
does **not** reproduce in 40 iterations. The same post-burst sequence still leaves the
channel claiming READY while the picked transport is dead, but the RPC fails fast
with `UNAVAILABLE` and gRPC immediately schedules a resolver refresh, after which the
channel recovers (`logs/run3-newpf-no-wedge/`).

So the catastrophic silent wedge appears specific to the legacy
`PickFirstLoadBalancer`. Since the new PF also has a known stuck-in-IDLE issue
(#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting
that PR — fixing #12395 and graduating the new PF would retire both wedge variants,
whereas today users are stuck choosing between a legacy PF with this READY wedge and
an experimental PF with a known IDLE wedge.

### Related issues

- #12395 — `PickFirstLeafLoadBalancer` stuck in IDLE on backend address change (Java
analog of grpc/grpc-go#8615). Same family: LB internal state diverging from the
published picker when a resolver update races a subchannel state change. Differs
in that it wedges in IDLE and affects the new leaf PF; this report is the legacy
PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and
no observable state ever changes).
- #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
- #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in
1.66); shows the resolver-error path mutating LB state non-atomically.

### Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until
process restart) during a rolling upgrade. The server saw zero incoming RPCs during
the window while the client channel claimed READY throughout. We now run an
application-level watchdog that calls `enterIdle()` when RPCs time out while the
channel claims READY.

### Attachment

`grpc-java-pickfirst-wedge.tar.gz` — repro sources, driver, and FINEST channel logs
for all three runs described above.

Start with io.grpc.internal.PickFirstLoadBalancer and run the attached BurstRepro.java, driver script, and proxy to reproduce the READY and CONNECTING wedges. Compare the legacy behavior with the PickFirstLeafLoadBalancer logs in logs/run2-READY-signature/ and logs/run3-newpf-no-wedge/. Done means the legacy policy no longer leaves a READY channel with a dead picker, and the repro's wait-for-ready probe either recovers or fails visibly.

Written by the indexing model from the issue text.