api-gateway xDS ships incomplete snapshots: clusters without endpoints, EDS never finishes

#### Overview of the Issue

A Consul native `api-gateway` with one HTTP listener and one `http-route` (~1545 backends) does not deliver a complete xDS snapshot.

Envoy gets **~279 partial CDS updates over ~100 minutes**. About **25% of clusters never receive EDS**, even though those services have healthy Connect instances. The gateway is `LIVE` during that window and returns `no healthy upstream` (503).

The same catalog, same Envoy 1.38.2 FIPS, on `ingress-gateway` converges in **2.0 seconds** (1546/1547 clusters, 1 CDS apply, `warming_clusters=0`). All 391 api-gateway misses got endpoints on ingress-gateway at t=2.002s.

#### Expected Behavior

`api-gateway` should push clusters with their endpoints (or an empty `ClusterLoadAssignment`) in one snapshot, the way `ingress-gateway` does for the same services.

#### Actual Behavior

| | `ingress-gateway` | `api-gateway` |
|---|---|---|
| Clusters | 1547 | 1545 |
| With endpoints | **1546 (99.9%)** | **1158 (75%)** |
| Never resolved | 1 (synthetic placeholder) | **391** |
| Time to endpoints | **2.0 s** | **~101 min, then stuck** |
| CDS applies | **1** | **279** |
| CDS interval | n/a | locks to **15.0 s** |

api-gateway phases (same Envoy process, `control_plane.connected_state=1` throughout):

1. **0–2918s** — CDS ramp: 40 → 1545 clusters; endpoints stay ~0
2. **2918–6045s** — EDS drip: 10 → 1144 clusters with endpoints (~0.36/s)
3. **6045–8004s** — tail: 1144 → 1158 (~0.007/s); 387 still empty

Those never-resolved clusters got no EDS — not even empty. Envoy waited `initial_fetch_timeout` (15s), then activated with 0 hosts. That is the 15.0s CDS cadence.

#### Suspected cause

`api-gateway` treats every `http-route` backend as a first-class upstream (discovery-chain watch + health watch). Consul does not wait for all watches to finish. As soon as one returns, Consul dirties its xDS snapshot and pushes CDS to Envoy — still missing the rest of the clusters/endpoints.

Catalog / EDS events appear to **recompile the full discovery-chain set** (`recompileDiscoveryChains`) rather than patch one cluster. CDS apply is Envoy-main-thread (measured 0.3–1.2s). EDS waits. Next catalog event starts another full CDS rebuild.

But `ingress-gateway` watches **one** chain and ships one complete snapshot. Same N backends, no drip.

---

#### Reproduction Steps

1. Consul 2.0.3 service mesh (Nomad/VM), ~1500 Connect HTTP services with passing instances.
2. Create `Kind=api-gateway`, one HTTP listener, one `http-route` with those services as backends.
3. Start `consul connect envoy -gateway api -register`.
4. Scrape Envoy admin `/stats`: `cluster_manager.active_clusters`, `warming_clusters`, `cds.update_success`, and host counts per cluster.
5. Repeat with `ingress-gateway` on the **same catalog**.

#### Operating system and Environment details

- Consul **2.0.3**
- Envoy **1.38.2 FIPS** (Consul-managed gateway)
- Nomad, Linux, Consul service mesh
- One `api-gateway`, one HTTP listener, TLS via `file-system-certificate`
- `consul connect envoy -gateway api -register`

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 1
check_ttls = 7
checks = 144
services = 103
build:
prerelease =
revision = d0f2be93
version = 2.0.3
version_metadata =
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 16
goroutines = 10937
max_procs = 16
os = linux
version = go1.26.6-X:boringcrypto
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1141
failed = 5
health_score = 0
intent_queue = 0
left = 0
member_time = 95942
members = 190
query_queue = 0
query_time = 19
```

```
Client agent HCL config
```

Server info

```
agent:
check_monitors = 1
check_ttls = 1
checks = 8
services = 11
build:
prerelease =
revision = d0f2be93
version = 2.0.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.14.88.44:8300
server = true
raft:
applied_index = 526873242
commit_index = 526873242
fsm_pending = 0
last_contact = 0
last_log_index = 526873242
last_log_term = 1166
last_snapshot_index = 526857016
last_snapshot_term = 1165
latest_configuration = [{Suffrage:Voter ID:b3250461-3187-7d47-c09d-50b9cdbc3689 Address:10.14.88.80:8300} {Suffrage:Voter ID:b0d0bd8b-bb20-b5a5-6fd3-a14c3429b2f3 Address:10.14.88.44:8300} {Suffrage:Voter ID:72a0085d-dc8e-1903-9e0d-a3057b998285 Address:10.14.88.56:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 1166
runtime:
arch = amd64
cpu_count = 16
goroutines = 56884
max_procs = 16
os = linux
version = go1.26.6-X:boringcrypto
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1143
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 95950
members = 185
query_queue = 0
query_time = 19
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1485
members = 3
query_queue = 0
query_time = 1
```

```
Server agent HCL config
```

### Log Fragments

Profiler summaries from Envoy admin `/stats`.

**ingress-gateway** (same catalog):

```
samples : 17 over 32s
clusters (peak) : 1547
clusters with endpoints : 1546 (99.9%)
time to 50%/90%/99%/100% of peak : 2s
endpoint attach rate : 48.189 clusters/sec
CDS applies observed : 2
CDS interval min/med/max: 0.83s / 0.83s / 0.83s
cluster_added / modified: 0 / 0
warming_clusters max / last: 0 / 0
```

Timeline:

```
elapsed clusters with_endpoints healthy_eps warming cds_ok
0.0 1547 3 3 0 1
2.0 1547 1546 2063 0 2
4.0 1547 1546 2063 0 2
... flat through 32s
```
**api-gateway** (2072 samples over 8004s):

```
clusters (peak) : 1545
clusters with endpoints : 1158 (75.0%)
time to 50% of peak : 4432s (74m)
time to 90% of peak : 5682s (95m) [measured]
time to 99% of peak : 6297s (105m)
endpoint attach rate : 0.145 clusters/sec (mean)
phase 1 CDS ramp : 0-> 2918s with_eps 3-> 10 0.0024 /s
phase 2 EDS drip : 2918-> 6045s with_eps 10-> 1144 0.3627 /s
phase 3 tail : 6045-> 8004s with_eps 1144-> 1158 0.0071 /s
CDS applies observed : 293
CDS interval min/med/max: 0.86s / 14.48s / 30.63s (phase 1, n=279)
cluster_added : 40 -> 1545 (+1505, incremental)
cluster_removed : 0
cluster_updated : 0 -> 1151 (= endpoint deliveries; with_eps - updated == 3 always)
warming_clusters max : 73 (non-zero in 99–100% of phase-1 samples)
control_plane.rate_limit_enforced: 0
control_plane.connected_state : 1 throughout
```

[alloc_logs_api-gateway.stderr.0.txt](https://github.com/user-attachments/files/31748867/alloc_logs_api-gateway.stderr.0.txt)
[api-gateway-xds-profile.json](https://github.com/user-attachments/files/31748919/api-gateway-xds-profile.json)
[api-gateway-xds-profile.ndjson.summary.txt](https://github.com/user-attachments/files/31748931/api-gateway-xds-profile.ndjson.summary.txt)
[api-gateway-xds-profile.ndjson.convergence.csv](https://github.com/user-attachments/files/31748948/api-gateway-xds-profile.ndjson.convergence.csv)

[ingress-gateway-xds-profile.json](https://github.com/user-attachments/files/31749001/ingress-gateway-xds-profile.json)
[ingress-gateway-xds-profile.ndjson.summary.txt](https://github.com/user-attachments/files/31749018/ingress-gateway-xds-profile.ndjson.summary.txt)
[ingress-gateway-xds-profile.ndjson.convergence.csv](https://github.com/user-attachments/files/31749027/ingress-gateway-xds-profile.ndjson.convergence.csv)

Start at the api-gateway xDS path used by `consul connect envoy -gateway api -register`, especially the `recompileDiscoveryChains` path and its discovery-chain and health watches. Compare its snapshot behavior with the ingress-gateway path using the supplied Envoy stats and profiles. Done means the api-gateway delivers complete clusters with endpoints, or explicit empty assignments, in a converged snapshot without the prolonged CDS/EDS drip.

Written by the indexing model from the issue text.