dev: make discovery failover atomic and observable

Summary

Harden endpoint discovery and transport recovery for long-running PHP processes. A successful discovery refresh currently updates the endpoint pool, but it does not migrate existing Table/Session service clients. After a transport failure, the SDK may also construct a new stub that reuses the same persistent PHP gRPC channel.

This issue is about making the recovery contract deterministic and observable. It does not prescribe force_new on every request or claim that client behavior is the only possible cause of an outage.

Confirmed behavior in v1.16.3

A successful refresh does not update existing service clients

Ydb::discover() applies the response to Cluster and updates the current endpoint value:

• Ydb::discover() and applyDiscoveryResult()
• Cluster::sync() removes endpoints absent from the latest response

However, Table creates one ServiceClient for the endpoint selected at construction time, and each Session copies that client:

• Table::__construct()
• Session::__construct()

Therefore, a successful scheduled refresh does not by itself move existing Table/Session clients away from an endpoint removed by the latest discovery response.

Error recovery can reuse the same persistent underlying channel

For every non-zero gRPC status, RequestTrait::handleGrpcStatus() performs discovery, randomly picks an endpoint, and constructs a new service stub without force_new:

• RequestTrait::handleGrpcStatus()

In PHP gRPC 1.73.0, channels with the same target and channel arguments are persistent. Without force_new, constructing a new stub may return the previously stored underlying channel:

• persistent-channel key
• force_new versus persistent-channel reuse

The bootstrap discovery implementation already handles this explicitly after a failed attempt: it closes the old client and creates a new one with force_new=true:

• Internal\Discovery::runRetryLoop() and recreateClient()

There is no equivalent explicit policy for Table/Session transport recovery. If random selection returns the same hostname, a new PHP stub does not guarantee a new underlying channel or a fresh DNS resolution.

Discovery scheduling is per service/session object

lastDiscovery and lastDiscoveryAttempt are fields of RequestTrait, which is used independently by Table, Session, Scheme, Scripting, Operations, and other objects:

• per-object discovery state
• checkDiscovery()

A newly created Session starts with zero-valued timers, so its first operation may trigger another ListEndpoints. Non-zero gRPC statuses also call discovery immediately outside the scheduled-refresh gate. Under many sessions/processes and correlated transport failures, this can amplify discovery traffic.

Current logs cannot reconstruct the recovery decision

The SDK does not log, in a structured way:

• the endpoint list returned by each refresh;
• the endpoint selected for a service client and why;
• a discovery/channel generation;
• whether a new underlying channel was forced;
• whether the selected endpoint had just disappeared from the latest discovery result.

This makes it difficult to distinguish a stale recursive resolver, a persistent gRPC channel, an old service client, and a valid selection of a temporarily unhealthy backend.

The documented contract says that discovery keeps the list fresh, balances subsequent requests, and re-discovers when a node becomes unavailable:

• README: Discovery

Expected behavior

For a long-running process with discovery => true:

1. Removing an endpoint from a successful discovery result should cause existing service clients to stop using it within a bounded time.
2. Recovery after a transport failure should not immediately reuse a just-failed endpoint/channel unless that is an explicit, tested policy.
3. Service and Session objects belonging to one Ydb instance should share or deduplicate discovery refresh state.
4. Diagnostic logs should show the chain bootstrap target -> discovery result -> selected endpoint -> channel generation/recreation -> RPC result, without credentials or other secrets.
5. Recovery should work with normal DNS TTL behavior and with a delayed recursive resolver.

The implementation may use generation-aware clients, explicit channel invalidation/close, temporary endpoint quarantine, a shared discovery coordinator, or another design. The observable contract and tests are more important than prescribing one mechanism.

Acceptance tests

1. A long-lived Session continues after its endpoint disappears from the next successful ListEndpoints result.
2. The A record of the same hostname changes after a transport failure; recovery does not remain indefinitely on the old persistent channel.
3. Of two endpoints, one shuts down and the other becomes ready; requests converge on the working endpoint without restarting the PHP process.
4. Correlated gRPC failures from multiple Session objects produce a bounded number of ListEndpoints calls rather than an independent refresh per object/request.
5. Debug logs expose the returned endpoint set, selected endpoint, and channel-generation/recreation decision without exposing authentication data.

Related issues

• #261
• #83
• #61

This client-side hardening cannot provide availability when no published backend is healthy; server-side rollout overlap remains a separate requirement.

Start with Ydb::discover(), Cluster::sync(), Table::__construct(), Session::__construct(), RequestTrait::handleGrpcStatus(), and checkDiscovery(); compare their behavior with Internal\Discovery::runRetryLoop() and recreateClient(). Use the five acceptance tests as the definition of done: clients recover from removed endpoints, channels are recreated when needed, refreshes are bounded, and debug logs expose decisions without credentials.