on demand CDS result in 503 for the 1st request
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
When a new cluster subscribes to an EDS resource (ClusterLoadAssignment) that is already subscribed by an existing cluster on the same ADS stream, the new cluster stalls in the warming state until FetchTimedout(15s) occurs.
We've observed a HTTP OnDemand filter timeout(default 5s), downstream requests waiting for cluster warming fail with HTTP 503 cluster_not_found, even though valid endpoints for that EDS resource are already cached in memory.
Root Cause Analysis
Deduplication suppresses wire requests: When the new cluster initiates its subscription via subscription_->start({edsServiceName()}), [WatchMap::findAdditions()](https://github.com/envoyproxy/envoy/blob/main/source/extensions/config_subscription/grpc/watch_map.cc) observes that the resource name is already present in watch_interest_ (from the existing cluster). Consequently, newly_added_to_subscription is empty, and NewGrpcMuxImpl does not send a DeltaDiscoveryRequest over the wire.
Control plane sends no push: Because no wire request is sent and the control plane is event-driven (only pushing deltas when endpoints change), the server sends no response.
Timer stalls for 15 seconds: [GrpcSubscriptionImpl::start()](https://github.com/envoyproxy/envoy/blob/main/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc) unconditionally arms a 15-second init_fetch_timeout_timer_. Since WatchMap does not prime the new watch with the cached resource sitting in EdsResourcesCache, the subscription waits for the full 15-second duration.
Premature client rejection (HTTP 503): At $T = 5\text{ s}$, the downstream HTTP filter's OnDemand timer expires, and Envoy returns a local reply with 503 Service Unavailable (cluster_not_found).
Delayed fallback: At $T = 15\text{ s}$, the EDS fetch timer expires with FetchTimedout, and [EdsClusterImpl::onConfigUpdateFailed](https://github.com/envoyproxy/envoy/blob/main/source/extensions/clusters/eds/eds.cc) finally queries EdsResourcesCache and transitions the cluster to active—10 seconds too late for the client request.
Steps to Reproduce
Configure a static cluster (e.g. cluster_a) using EDS resource xdstp://.../endpoint_x.
Start Envoy. Observe that cluster_a receives endpoints at startup and populates EdsResourcesCache.
Send a downstream request that triggers On-Demand CDS for a dynamic cluster (cluster_b) whose eds_cluster_config.service_name also points to xdstp://.../endpoint_x.
Observe the log timeline:
$T = 0\text{ s}$: CDS response arrives; cluster_b starts warming. No EDS discovery request is queued on the gRPC stream.
$T = 5\text{ s}$: cm odcds: on-demand discovery for cluster ... timed out -> 503 returned to downstream client.
$T = 15\text{ s}$: gRPC config: initial fetch timed out for ... ClusterLoadAssignment -> Did not receive EDS response on time, using cached ClusterLoadAssignment -> cluster warming completes.
Expected Behavior
When a new watch is registered for an EDS resource that is already active and cached in EdsResourcesCache, Envoy should immediately prime the new watch from the cache at $T = 0$, disarm the init_fetch_timeout_timer_, and complete cluster warming without waiting for an unsolicited wire response.
Contributor guide
Research direction
Start by tracing WatchMap::findAdditions() in source/extensions/config_subscription/grpc/watch_map.cc and GrpcSubscriptionImpl::start() in grpc_subscription_impl.cc. Compare this path with EdsClusterImpl::onConfigUpdateFailed() in source/extensions/clusters/eds/eds.cc and its EdsResourcesCache fallback. Done means a new cached EDS watch completes warming immediately without waiting for the fetch timer or producing the downstream 503.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100