Graceful client shutdown

Is your feature request related to a problem? Please describe.

It appears that the hyper-util client isn’t currently gracefully shutting down its connection with the server. Specifically, the hyper client seems to be calling shutdown on the client’s stream, and then immediately closing the socket without waiting for the server’s shutdown. This can result in a NotConnected socket error when the server attempts to send its shutdown to the client, since the socket is already closed.

Describe the solution you'd like

It would be useful to have a way to allow the hyper-util client to gracefully shutdown, in order to avoid issues like the NotConnected socket error emitting on the server. This could be in the form of a Client shutdown() API, which sends a shutdown and waits for an end-of-stream indication from the server, before returning control to the application and closing the socket.

Additional context

The NotConnected socket error seems to be raised due to a socket race condition where:

• The client sends its shutdown and immediately closes its socket.
• The server writes some data to the socket
• The server attempts to send its shutdown to the client, raising the NotConnected error.

If the server doesn’t write any data to the socket before sending its shutdown, a socket error isn’t raised.

I encountered this issue while testing the s2n-tls hyper-util connector crate. I believe this issue is immediately observable for TLS streams since TLS streams will always write a close notify alert when sending a shutdown. However, there may be ways that this issue could impact normal TCP streams as well, such as if the server is sending data with HTTP server push before shutting down.

To verify that this isn’t just an issue with s2n-tls, I reproduced the issue with rustls (modified from the hyper-rustls examples). However, it seems that rustls works around this issue by ignoring NotConnected socket errors in tokio-rustls (https://github.com/rustls/tokio-rustls/issues/41). When I removed this suppression from poll_shutdown() in tokio-rustls, I observed the same NotConnected error that I’m seeing with s2n-tls-tokio. A possible solution to this problem could be to similarly ignore NotConnected errors in s2n-tls-tokio. However, it seems like it could be better for the hyper client to perform a graceful shutdown instead.

I was also able to reproduce this issue outside of a TLS stream by wrapping a TCP stream with a poll_shutdown method that writes some data to the socket before shutdown. See the following rust playground example:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=d4baf4675d8af0a45d0c924a933c7812

Start by tracing the hyper-util client shutdown path and its poll_shutdown() behavior. Compare the referenced tokio-rustls src/common/mod.rs suppression and the linked reproductions to understand the socket race. Done means the chosen shutdown behavior prevents the server-side NotConnected error while preserving the client API's expected completion semantics.