envoyproxy / envoyproxy/envoy

Question: Should client-side request timeouts contribute to outlier detection?

Open
#41,985 5 comments 0 reactions 0 assignees View on GitHub
area/outlier_detection no stalebot question
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

Hi Envoy team,

I noticed that when a request from Envoy's upstream cluster times out due to a client-side request timeout (e.g., in an A server → A Envoy → B Envoy → B server), these events are not counted in outlier detection metrics (consecutive_5xx, consecutive_local_origin_failure).

Current behavior (from /cluster stats):
`cx_total: 27
rq_active: 0
rq_error: 6
rq_success: 6
rq_timeout: 0
rq_total: 32`

In this case, the code 504 triggered by the client-side timeout are only reflected in rq_total, but not in any outlier detection counters.

Below is the outlier_detection configuration from my CDS cluster:
`"outlier_detection": {
"consecutive_5xx": 3,
"interval": "5s",
"base_ejection_time": "5s",
"max_ejection_percent": 100,
"enforcing_consecutive_5xx": 100,
"enforcing_success_rate": 0,
"success_rate_minimum_hosts": 5,
"success_rate_request_volume": 100,
"success_rate_stdev_factor": 1900,
"consecutive_gateway_failure": 3,
"enforcing_consecutive_gateway_failure": 100,
"split_external_local_origin_errors": true,
"consecutive_local_origin_failure": 2,
"enforcing_consecutive_local_origin_failure": 100,
"failure_percentage_threshold": 10,
"enforcing_failure_percentage": 100,
"enforcing_failure_percentage_local_origin": 100,
"failure_percentage_minimum_hosts": 0
}`

My questions:

1. Is this behavior expected — i.e., Envoy intentionally does not treat client-side request timeouts as “local origin failures”?
2. If not, is there any existing way for the caller-side Envoy (A) to detect such timeouts and automatically eject the problematic upstream host (B)?

If there is currently no mechanism, I’d like to propose adding a configurable option to treat client-side timeout cancellations as outlier-detectable failures — for example, contributing to consecutive_local_origin_failure or via a new metric type.

This would help Envoy automatically shield upstream hosts that are consistently slow to respond, improving resilience in high-QPS microservice environments where latency spikes often cause long-tail timeouts.

I’d be happy to discuss this further and open a PR if the approach makes sense.

envoy version : 1.33.6
Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.