envoyproxy / envoyproxy/envoy

on demand CDS result in 503 for the 1st request

Open
#47,302 3 comments 0 reactions 0 assignees View on GitHub
area/xds bug investigate
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

When a new cluster subscribes to an EDS resource (ClusterLoadAssignment) that is already subscribed by an existing cluster on the same ADS stream, the new cluster stalls in the warming state until FetchTimedout(15s) occurs.

We've observed a HTTP OnDemand filter timeout(default 5s), downstream requests waiting for cluster warming fail with HTTP 503 cluster_not_found, even though valid endpoints for that EDS resource are already cached in memory.

Root Cause Analysis
Deduplication suppresses wire requests: When the new cluster initiates its subscription via subscription_->start({edsServiceName()}), [WatchMap::findAdditions()](https://github.com/envoyproxy/envoy/blob/main/source/extensions/config_subscription/grpc/watch_map.cc) observes that the resource name is already present in watch_interest_ (from the existing cluster). Consequently, newly_added_to_subscription is empty, and NewGrpcMuxImpl does not send a DeltaDiscoveryRequest over the wire.
Control plane sends no push: Because no wire request is sent and the control plane is event-driven (only pushing deltas when endpoints change), the server sends no response.
Timer stalls for 15 seconds: [GrpcSubscriptionImpl::start()](https://github.com/envoyproxy/envoy/blob/main/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc) unconditionally arms a 15-second init_fetch_timeout_timer_. Since WatchMap does not prime the new watch with the cached resource sitting in EdsResourcesCache, the subscription waits for the full 15-second duration.
Premature client rejection (HTTP 503): At $T = 5\text{ s}$, the downstream HTTP filter's OnDemand timer expires, and Envoy returns a local reply with 503 Service Unavailable (cluster_not_found).
Delayed fallback: At $T = 15\text{ s}$, the EDS fetch timer expires with FetchTimedout, and [EdsClusterImpl::onConfigUpdateFailed](https://github.com/envoyproxy/envoy/blob/main/source/extensions/clusters/eds/eds.cc) finally queries EdsResourcesCache and transitions the cluster to active—10 seconds too late for the client request.
Steps to Reproduce
Configure a static cluster (e.g. cluster_a) using EDS resource xdstp://.../endpoint_x.
Start Envoy. Observe that cluster_a receives endpoints at startup and populates EdsResourcesCache.
Send a downstream request that triggers On-Demand CDS for a dynamic cluster (cluster_b) whose eds_cluster_config.service_name also points to xdstp://.../endpoint_x.
Observe the log timeline:
$T = 0\text{ s}$: CDS response arrives; cluster_b starts warming. No EDS discovery request is queued on the gRPC stream.
$T = 5\text{ s}$: cm odcds: on-demand discovery for cluster ... timed out -> 503 returned to downstream client.
$T = 15\text{ s}$: gRPC config: initial fetch timed out for ... ClusterLoadAssignment -> Did not receive EDS response on time, using cached ClusterLoadAssignment -> cluster warming completes.
Expected Behavior
When a new watch is registered for an EDS resource that is already active and cached in EdsResourcesCache, Envoy should immediately prime the new watch from the cache at $T = 0$, disarm the init_fetch_timeout_timer_, and complete cluster warming without waiting for an unsolicited wire response.

Contributor guide

Open the contributing guide

Research direction

Start by tracing WatchMap::findAdditions() in source/extensions/config_subscription/grpc/watch_map.cc and GrpcSubscriptionImpl::start() in grpc_subscription_impl.cc. Compare this path with EdsClusterImpl::onConfigUpdateFailed() in source/extensions/clusters/eds/eds.cc and its EdsResourcesCache fallback. Done means a new cached EDS watch completes warming immediately without waiting for the fetch timer or producing the downstream 503.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.