envoyproxy / envoyproxy/envoy

Envoy HTTP2 Connection sends RST_STREAM before initial HEADERS which causes upstream server to close connection with PROTOCOL_ERROR

Open
#11,369 20 comments 0 reactions 1 assignee Assigned to @antoniovicente View on GitHub
area/http bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

### Description
Preface: we’re using envoy as an L7 proxy with a HTTP2 upstream. We’ve noticed that sometimes Envoy sometimes sends a RST_STREAM before actually sending the HEADERS frame to change the stream state from idle to open.

The following is a pcap of a single TCP connection between envoy and the upstream server. The upstream server is listening on 8080 while the ephemeral port that envoy cx is using is 63692.

[Pcap file](https://github.com/envoyproxy/envoy/files/4703497/63692.pcapng.zip)

![image](https://user-images.githubusercontent.com/43147603/83290606-b9f40000-a19b-11ea-9399-25079cfbe343.png)

On `#243`, we can see that envoy starts sending RST_STREAM for streams that haven’t received HEADERS or PUSH-PROMISE frames. Additionally, some of these stream resets are repeated (e.g. stream id 27, 23 from above). Because of this the server closes the connection with a PROTOCOL_ERROR as per [HTTP2 spec](https://http2.github.io/http2-spec/#RST_STREAM).

Excerpt:
> RST_STREAM frames MUST NOT be sent for a stream in the "idle" state. If a RST_STREAM frame identifying an idle stream is received, the recipient MUST treat this as a connection error (Section 5.4.1) of type PROTOCOL_ERROR.

This leads to envoy dropping in-flight requests on the same HTTP2 connection with a 503 UC. We believe this is because of the way we do deferred reset after reading this code [snippet](https://github.com/envoyproxy/envoy/blob/master/source/common/http/http2/codec_impl.cc#L933). In other words, we believe that from envoy’s prospective, this stream exists, but since the first frame hasn’t made it to the server yet, the server believes that the stream is idle and does an active close.

### Repro Steps
A very contrived example can be done using a single upstream go service coupled with a load tester and envoy.

**Version details:**
* Go 1.14.2
* envoyproxy/envoy:v1.14.1 docker image
* Hey - 0.1.3

**Configuration:**
* HTTP2 Server - https://gist.github.com/tan-stripe/4ea1e8f3dfbb3af6210f655dc2883647
* envoy.yaml - https://gist.github.com/tan-stripe/8965f551d7ffe68cdb4c4d738d143a63
* Dockerfile - https://gist.github.com/tan-stripe/d6bfb04c5ad8c044442b0aefa07d898b

**Steps:**
1. Start up a simple HTTP2 server with some delay in the response
2. Start envoy with given configuration
3. Run a hey loadtest - `hey -n 50 -c 50 -h2 -t 3 http://localhost:10000`
* You may have to tune the parameters a little bit, but we were able to reliably cause an error message on the go program side. Error: http2: server connection error from 127.0.0.1:65100: connection error: PROTOCOL_ERROR
* This error comes from receiving a RST_STREAM on an idle stream
* Any connection on the same TCP connection gets dropped due to a TCP-FIN initiated by the go server

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.