Issues with client termination of H2 CONNECT streams

Version
Hyper 1.3

Platform

6.8.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 15:20:28 +0000 x86_64 GNU/Linux

Description

We are seeing a few issues around using H2 CONNECT. Our application is using Tokio, Rustls, and Hyper to communicate between a client and server both using the same stack. We have multiple streams on one TCP connection sometimes (though the issue occurs regardless of multiplexing).

Our first issue that popped up was leaking connections. This was debugging and an (attempted) fix was opened in https://github.com/hyperium/hyper/pull/3647. More details there. This was like not detected before because:

• The CONNECT tests in the repo all close from the server side
• When using hyper's pool this isn't seen since the pool will hold onto the connection after all streams are closed (until the idle timeout). We only started seeing this when we stopped using hyper's pool and started closing the connection immediately after the last stream closed.

With that fix, everything was working fine. However, a later refactor in our broke things again; we started dropping the SendRequest before we were complete with IO operations on Upgraded (before, we dropped the SendRequest after). This causes the stream/connection to be closed unexpectedly. This can be reproduced easily by moving the drop(client) in https://github.com/hyperium/hyper/pull/3647 to before the write/read operations. This is easy to workaround, but its not documented and @seanmonstar suggested it was not expected on Discord.

Start with the CONNECT tests mentioned in the issue and the attempted fix in PR #3647, then reproduce the behavior by dropping SendRequest before the Upgraded read/write operations. Done means client-side termination no longer unexpectedly closes the stream or leaks the connection, with coverage for the reported ordering.