Canceled streams are not cleaned up in time
- Dominant language
- C++
- Stars
- 2k
- Forks
- 874
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 46
Description
We run a proxy server in front of ATS and noticed this problem when the number of streams on a connections gets close to the maximum, and some streams are canceled by the proxy (generally because the original request was canceled).
I posted some test code and logs of a reproduction case in [this gist](https://gist.github.com/mrdg/240bb61a0df2b17b8cc9148c4d540ed9). I tested this on master with `proxy.config.http2.max_concurrent_streams_in` set to 5 to make it easier to parse the logs, but we see the issue with much higher limits as well. The test client sends 5 requests, then cancels those, and then sends 5 more, which should be accepted, but it appears that ATS is refusing them. The client code logs this:
```
2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=11 len=4 ErrCode=REFUSED_STREAM
2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=13 len=4 ErrCode=REFUSED_STREAM
2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=15 len=4 ErrCode=REFUSED_STREAM
2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=17 len=4 ErrCode=REFUSED_STREAM
2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=19 len=4 ErrCode=REFUSED_STREAM
```
Looking at the ATS logs, I see that the previously canceled streams are deleted after the second batch of requests is rejected:
```
[Nov 4 12:38:26.285] [ET_NET 1] DEBUG: (http2_cs) [1] receiving frame header
[Nov 4 12:38:26.285] [ET_NET 1] DEBUG: (http2_cs) [1] frame header length=10, type=1, flags=0x5, streamid=11
[Nov 4 12:38:26.285] [ET_NET 1] DEBUG: (http2_con) [1] [11] Received HEADERS frame
[Nov 4 12:38:26.286] [ET_NET 1] DEBUG: (http2_con) [1] [11] Send RST_STREAM frame
...
[Nov 4 12:38:26.288] [ET_NET 1] DEBUG: (http2_stream) [1] [1] Destroy stream, sent 0 bytes
[Nov 4 12:38:26.288] [ET_NET 1] DEBUG: (http2_con) [1] [1] Delete stream
```
We also see this in the logs:
```
[Nov 4 12:38:26.287] [ET_NET 1] ERROR: HTTP/2 stream error code=0x07 client_ip=127.0.0.1 session_id=1 stream_id=19 recv headers creating inbound stream beyond max_concurrent limit
```
We also see this compression error (or sometimes a protocol error) when the problem happens:
```
[Nov 4 12:38:26.288] [ET_NET 1] ERROR: HTTP/2 connection error code=0x09 client_ip=127.0.0.1 session_id=1 stream_id=21 recv headers compression error
```
Contributor guide
Assessment
This issue has not been assessed yet.