pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

[grpc-java-pickfirst-wedge.tar.gz](https://github.com/user-attachments/files/30553073/grpc-java-pickfirst-wedge.tar.gz)

### What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

### What is your environment?

Linux, Java 17 (also observed under Java 11 in production).
Load balancing policy: `pick_first` (single-address target).
Active LB implementation: legacy `io.grpc.internal.PickFirstLoadBalancer` (its
`RequestConnectionPicker` appears in the channel logs; the new
`PickFirstLeafLoadBalancer` was not enabled — see "Additional data point" below).

### What did you expect to see?

After a transient network failure heals, a `pick_first` channel either recovers or
transitions out of READY so that the LB policy or the application (via
`notifyWhenStateChanged`) can react.

### What did you see instead?

The channel is left **permanently reporting READY** while every RPC — including
`withWaitForReady()` RPCs — buffers forever and fails only by its own deadline:

```
DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]
```

No channel state transition ever fires again, so nothing can observe that the channel
is broken. `ManagedChannel.enterIdle()` reliably recovers it.

Mechanism, from FINEST `ChannelLogger` logs (attached):

1. The transport to the current address dies (TLS handshakes reset — in production,
a rolling container upgrade left the port published while the backend reset
handshakes for ~10 seconds).
2. gRPC's reconnect logic calls `NameResolver.refresh()` repeatedly. The resolver
answers asynchronously with a mix of `onError` and `onResult` deliveries (it is
backed by a discovery service that sits behind the same failing network, so
deliveries arrive out of order relative to subchannel state changes).
3. Each `onResult` causes `PickFirstLoadBalancer` to create a replacement subchannel
and schedule delayed shutdown (~5s) of the previous one.
4. Within the delayed-shutdown window, the old subchannel completes its connection
against the recovering backend and reaches READY; the LB publishes a READY picker
referencing it.
5. The delayed shutdown then fires (`SHUTDOWN with UNAVAILABLE(Subchannel shutdown
invoked)`), terminating the transport — but the published READY picker is never
replaced. The channel stays READY forever with a picker whose subchannel has no
transport.

### Steps to reproduce the bug

Self-contained repro attached: one Java file (`BurstRepro.java`, only grpc-java +
netty dependencies, in-process backend server), a small Python TCP proxy that models
the network failure (`reset` mode: accept + immediately close, emulating TLS resets),
and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to `reset`
for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy,
then probe with a `withWaitForReady()` RPC with a deadline.

Two wedge variants observed:

- `logs/run2-READY-signature/` — channel READY, wait-for-ready probe black-holes to
`DEADLINE_EXCEEDED`; `enterIdle()` immediately recovers it. This matches our
production incident signature.
- `logs/run1-connecting-wedge/` — channel wedged in CONNECTING with no reconnect
attempt scheduled.

### Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with `GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true`
(confirmed active: `PickFirstLeafLoadBalancer` in the logs), the permanent black-hole
does **not** reproduce in 40 iterations. The same post-burst sequence still leaves the
channel claiming READY while the picked transport is dead, but the RPC fails fast
with `UNAVAILABLE` and gRPC immediately schedules a resolver refresh, after which the
channel recovers (`logs/run3-newpf-no-wedge/`).

So the catastrophic silent wedge appears specific to the legacy
`PickFirstLoadBalancer`. Since the new PF also has a known stuck-in-IDLE issue
(#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting
that PR — fixing #12395 and graduating the new PF would retire both wedge variants,
whereas today users are stuck choosing between a legacy PF with this READY wedge and
an experimental PF with a known IDLE wedge.

### Related issues

- #12395 — `PickFirstLeafLoadBalancer` stuck in IDLE on backend address change (Java
analog of grpc/grpc-go#8615). Same family: LB internal state diverging from the
published picker when a resolver update races a subchannel state change. Differs
in that it wedges in IDLE and affects the new leaf PF; this report is the legacy
PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and
no observable state ever changes).
- #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
- #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in
1.66); shows the resolver-error path mutating LB state non-atomically.

### Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until
process restart) during a rolling upgrade. The server saw zero incoming RPCs during
the window while the client channel claimed READY throughout. We now run an
application-level watchdog that calls `enterIdle()` when RPCs time out while the
channel claims READY.

### Attachment

`grpc-java-pickfirst-wedge.tar.gz` — repro sources, driver, and FINEST channel logs
for all three runs described above.

io.grpc.internal.PickFirstLoadBalancer から始め、添付されている BurstRepro.java、ドライバースクリプト、プロキシを実行して READY と CONNECTING のスタックを再現します。legacy の動作を、logs/run2-READY-signature/ と logs/run3-newpf-no-wedge/ にある PickFirstLeafLoadBalancer のログと比較します。legacy policy が dead picker を持つ READY channel を残さなくなり、repro の wait-for-ready probe が復旧するか、目に見える形で失敗すれば完了です。

索引モデルが issue の本文から書いたものです。