Question: Should client-side request timeouts contribute to outlier detection?
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
Hi Envoy team,
I noticed that when a request from Envoy's upstream cluster times out due to a client-side request timeout (e.g., in an A server → A Envoy → B Envoy → B server), these events are not counted in outlier detection metrics (consecutive_5xx, consecutive_local_origin_failure).
Current behavior (from /cluster stats):
`cx_total: 27
rq_active: 0
rq_error: 6
rq_success: 6
rq_timeout: 0
rq_total: 32`
In this case, the code 504 triggered by the client-side timeout are only reflected in rq_total, but not in any outlier detection counters.
Below is the outlier_detection configuration from my CDS cluster:
`"outlier_detection": {
"consecutive_5xx": 3,
"interval": "5s",
"base_ejection_time": "5s",
"max_ejection_percent": 100,
"enforcing_consecutive_5xx": 100,
"enforcing_success_rate": 0,
"success_rate_minimum_hosts": 5,
"success_rate_request_volume": 100,
"success_rate_stdev_factor": 1900,
"consecutive_gateway_failure": 3,
"enforcing_consecutive_gateway_failure": 100,
"split_external_local_origin_errors": true,
"consecutive_local_origin_failure": 2,
"enforcing_consecutive_local_origin_failure": 100,
"failure_percentage_threshold": 10,
"enforcing_failure_percentage": 100,
"enforcing_failure_percentage_local_origin": 100,
"failure_percentage_minimum_hosts": 0
}`
My questions:
1. Is this behavior expected — i.e., Envoy intentionally does not treat client-side request timeouts as “local origin failures”?
2. If not, is there any existing way for the caller-side Envoy (A) to detect such timeouts and automatically eject the problematic upstream host (B)?
If there is currently no mechanism, I’d like to propose adding a configurable option to treat client-side timeout cancellations as outlier-detectable failures — for example, contributing to consecutive_local_origin_failure or via a new metric type.
This would help Envoy automatically shield upstream hosts that are consistently slow to respond, improving resilience in high-QPS microservice environments where latency spikes often cause long-tail timeouts.
I’d be happy to discuss this further and open a PR if the approach makes sense.
envoy version : 1.33.6
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.