envoyproxy / envoyproxy/envoy

Intermittent 503 UC (upstream_reset_before_response_started{connection_termination})

Open
#40,468 10 comments 1 reaction 0 assignees View on GitHub
area/http area/http_connection_manager bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

This may be the same issue as #14981 which was never resolved, but now with more details.

The configuration in which these errors occur is:
1. http/1.1 request from downstream client to `envoy_edge` (supports http/2, but client didn't use it).
2. http/2 connection between `envoy_edge` and `envoy_a`.
3. http/1.1 request from `envoy_a` to upstream service.

The observed error is in the `envoy_edge` log, like (redacted to only the relevant parts):
```
{
"bytes_received": 0,
"bytes_sent": 1005,
"duration": 1001,
"response_code": 503,
"response_code_details": "upstream_reset_before_response_started{connection_termination}",
"response_flags": "UC", // UpstreamConnectionTermination
}
```
Noteworthy here is that duration is consistently in the range 1000-1003, indicating that a timeout is involved. The only timeout I could find that's 1 second is `delayed_close_timeout`, and experimentally verified that changing `delayed_close_timeout` in `envoy_edge` does not change the outcome, i.e. this duration is presumably the `delayed_close_timeout` of `envoy_a` (though this was not experimentally verified by changing that number).

Reconfiguring the connection between `envoy_edge` and `envoy_a` to only use http/1.1 resolves the problem and the errors no longer occur. So we can work around the problem, but this doesn't resolve the issue, which is that this is a misbehavior on envoy's part.

None of the observed cases featured a downstream client using http/2, so it seems *likely* that the trigger for this behavior involves some attribute of the request - though I think it's the request *before* the one that receives the error, that causes the issue, so this observation might be a red herring.

My hypothesis is that one or both of the request or response features some signal that triggers a connection half-close - either a `connection: close` header or a body with no content-length and not chunked encoding, which requires http/1.1 to half-close the connection to signal the end of the stream. Such a close is not supposed to propagate to the http/2 connection, but for this error to occur it seems that it must be. An idle timeout on the http/2 connection would be closing instantly, not after `delayed_close_timeout`, so it can't be a race against idle connections.

It may be relevant that this is observed in version 1.32.7, so has not yet picked up the switch from nghttp2 to oghttp2. It's not clear where in the chain the misbehavior occurs so it's possible that it's inside one of those libraries and may be resolved by that change. Unfortunately we can't really experiment with this, as the level of errors in production traffic is deemed unacceptable, so we have to keep the workaround in place.

The observed rate of errors is approximately 6 out of 2000 requests, but I assume this just depends on how often the upstream/downstream does whatever the triggering event is.

Metrics-wise, `downstream_cx_destroy` on `envoy_a` is suspiciously high, ~45 per second per instance, vs. a different envoy instance that doesn't have the same upstream service, `envoy_b` only does `downstream_cx_destroy` ~0.1 per second per instance.

Given these rates I would say that this might be *two* issues - one, that the http/2 connection is being closed when it shouldn't be, and two, that a half-close when it occurs is *supposed to* prevent the downstream from making more requests on the same connection, but there's a race in which it's possible for a request to arrive during the transition. Otherwise we'd be seeing 45 errors per second per instance (and some of them would have durations slightly less than 1 second).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.