Extension manager can stall xDS translation indefinitely on extension server or Kubernetes API outage

*Description*:

[internal/extension/registry/extension_manager.go](https://github.com/envoyproxy/gateway/blob/0e2610695aa5dea108198660d1c25f054f6f1ce1/internal/extension/registry/extension_manager.go) has several code paths that use context.Background() without any deadline, combined with blocking I/O. When the upstream being waited on is unreachable, these paths block indefinitely. Because the xDS translator calls extension hooks synchronously on the
translate path, a single blocked call stalls the whole xDS snapshot update — Envoy never receives new snapshots, and FailOpen does not help because it only fires on returned errors; these calls never return.

There are two independent scenarios:

1. RPC hang when the extension server is unreachable. Every hook method in [internal/extension/registry/xds_hook.go](https://github.com/envoyproxy/gateway/blob/28859d921ee496ef0c539aad14fe4b452c48eb74/internal/extension/registry/xds_hook.go) (e.g. PostRouteModifyHook, PostHTTPListenerModifyHook, …) invokes the gRPC stub with ctx := context.Background(). The service config generated by setupGRPCOpts sets "waitForReady": true
(extension_manager.go:46), so when the server is down the subchannel stays in TRANSIENT_FAILURE and the RPC is queued at the channel indefinitely. The retryPolicy does not kick in — retry attempts only count dispatched RPCs, not wait-for-ready queueing.

2. TLS cert validation hang when the Kubernetes API is unreachable (only applies when ExtensionService.TLS is set). Two sub-paths:
- Startup: setupGRPCOpts calls getCertPoolFromSecret(ctx, ...) (line 303) and getClientCertificateFromSecret(ctx, ...) (line 310) with a bare context.Background() from GetPreXDSHookClient / GetPostXDSHookClient. Both terminate at client.Get(ctx, ...) in internal/kubernetes/secret.go:34.
- Runtime: createGetRootCertificatesHandler (line 369) and createGetClientCertificatesHandler (line 399) return closures that hardcode ctx := context.Background() (lines 371, 401). Per the google.golang.org/grpc/security/advancedtls godoc, these are invoked "for every new connection" — i.e. on every
TLS handshake. No rest.Config.Timeout is configured anywhere in the repo (newK8sClient at [extension_manager.go:68](https://github.com/envoyproxy/gateway/blob/0e2610695aa5dea108198660d1c25f054f6f1ce1/internal/extension/registry/extension_manager.go#L68) uses GetConfigOrDie() with no override), so the Kubernetes reads are bounded only by TCP-level timeouts (minutes on Linux).

Expected behaviour: A hook RPC or a TLS cert refresh against an unreachable backend should fail within a bounded time (either DeadlineExceeded or UNAVAILABLE). The caller should then decide what to do based on FailOpen. xDS translation should never stall indefinitely on a single unreachable dependency.

Scenario 2 (runtime handshake callbacks) is particularly severe because it is a lifetime-of-process risk and is not fixed by adding a deadline at the RPC boundary — the callback runs synchronously inside the gRPC transport and does not see the RPC context.

*Repro steps*:

Scenario 1 (gRPC RPC hang — no TLS required):

1. Deploy envoy-gateway with an EnvoyGateway config that registers an ExtensionManager:
```
extensionManager:
service:
host: 127.0.0.1
port: 65432 # anything that will refuse connections
hooks:
xdsTranslator:
post: [Route]
```
2. Apply an HTTPRoute whose translation path reaches processExtensionPostRouteHook (internal/xds/translator/extension.go:31).
3. Observe that translation for that route never completes — no snapshot update is published. Goroutine dumps show *XDSHook.PostRouteModifyHook blocked in grpc.(*ClientConn).Invoke waiting for waitForReady.

A minimal unit reproducer is also included as the test Test_Integration_PostHook_UnreachableServerMustNotHang in internal/extension/registry/extension_manager_test.go on this branch. It reserves then releases a TCP port, points an ExtensionManager at that dead port, and asserts the RPC returns within
3s. It fails today with a timeout message.

Scenario 2 (TLS cert validation hang):

1. Configure an ExtensionManager with service.tls.certificateRef pointing at any Secret.
2. Disrupt the in-cluster API server (or block the API server IP with iptables, or point KUBECONFIG at a sink address).
3. 2a — on process startup (first hook invocation), setupGRPCOpts hangs in client.Get.
4. 2b — with a running process, forcibly terminate the existing extension server connection so gRPC re-handshakes. The handshake's GetRootCertificates / GetIdentityCertificatesForClient callback hangs in client.Get for every new connection attempt.

*Environment*:

- envoy-gateway version: latest main
- envoy version: what main uses
- kubernetes version: any
- google.golang.org/grpc version (relevant to the gRPC behaviour): v1.80.0 (from go.mod)
- platform: any (bug is in Go code, not platform-specific)

*Logs*:

From the Scenario 1 reproducer test run, gRPC's internal logs show the subchannel retrying TCP connects at its own cadence while the RPC stays queued:
```
W0421 22:37:09.277921 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:57843"}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:57843: connect: connection refused"
W0421 22:37:10.279751 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:57843"} ... connect: connection refused
W0421 22:37:11.656535 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:57843"} ... connect: connection refused
extension_manager_test.go:741: PostRouteModifyHook did not return within 3s — RPC is stuck waiting for a never-ready subchannel (waitForReady=true + context.Background() in XDSHook)
--- FAIL: Test_Integration_PostHook_UnreachableServerMustNotHang (3.00s)
```
The ~1.0s and ~1.4s gaps between connect attempts match the MaxBackoff: 1s in the generated service config (with jitter) — confirming these are subchannel-level reconnect attempts managed by gRPC's state machine, not RPC-level retries. The RPC itself never gets an attempt dispatched for MaxAttempts to
count against.

Envoy access logs are not relevant for this bug (no Envoy requests are affected directly — it's the control plane → Envoy snapshot update that stalls).

Test:

```
// Test_Integration_PostHook_UnreachableServerMustNotHang is a regression
// test for the waitForReady + context.Background() interaction in
// XDSHook. The service config sets waitForReady=true and every hook
// method in xds_hook.go calls the gRPC stub with context.Background(),
// so when the extension server is unreachable the RPC is queued on the
// subchannel forever and the retry policy never fires (retries count only
// dispatched attempts, not wait-for-ready queueing).
//
// The test asserts the *desired* behavior: an RPC against a down server
// returns an error within a bounded time. It fails today; adding a
// deadline to the hook RPC (or disabling waitForReady) makes it pass.
func Test_Integration_PostHook_UnreachableServerMustNotHang(t *testing.T) {
// Reserve an ephemeral port, then release it so connects get refused.
lis, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
port := lis.Addr().(*net.TCPAddr).Port
require.NoError(t, lis.Close())

mgr := &Manager{
extension: egv1a1.ExtensionManager{
Hooks: &egv1a1.ExtensionHooks{
XDSTranslator: &egv1a1.XDSTranslatorHooks{
Post: []egv1a1.XDSTranslatorHook{egv1a1.XDSRoute},
},
},
Service: &egv1a1.ExtensionService{
BackendEndpoint: egv1a1.BackendEndpoint{
IP: &egv1a1.IPEndpoint{
Address: "127.0.0.1",
Port: int32(port),
},
},
},
},
}
t.Cleanup(mgr.CleanupHookConns)

// grpc.Dial must be non-blocking — GetPostXDSHookClient should
// return a client immediately even though the server is unreachable.
dialStart := time.Now()
client, err := mgr.GetPostXDSHookClient(egv1a1.XDSRoute)
dialElapsed := time.Since(dialStart)

require.NoError(t, err)
require.NotNil(t, client)
require.Less(t, dialElapsed, time.Second,
"GetPostXDSHookClient should return immediately; took %v", dialElapsed)

// The RPC must not block past this budget. The gRPC retry policy
// (MaxAttempts=4, MaxBackoff=1s) plus a small fudge factor puts a
// bounded failure well under 3s if the bug is fixed.
const rpcBudget = 3 * time.Second
done := make(chan error, 1)
go func() {
_, err := client.PostRouteModifyHook(&routev3.Route{Name: "r"}, nil, nil)
done <- err
}()

select {
case rpcErr := <-done:
require.Error(t, rpcErr, "expected an error when server is down")
case <-time.After(rpcBudget):
t.Fatalf("PostRouteModifyHook did not return within %v — "+
"RPC is stuck waiting for a never-ready subchannel "+
"(waitForReady=true + context.Background() in XDSHook)", rpcBudget)
}
}
```