`Change graceful_shutdown function behavior` PR can cause tonic servers to hang `serve_with_incoming_shutdown` in two different cases

Version

Foudn in hyper 1.5.1 and later

Platform

25.5.0 (macOS), though it is also applying on our linux bots

Summary

At OneSignal we have a number of tonic gRPC services we maintain. After a recent update of dependencies on one of them, we noticed that it "randomly" would hang during shutdown for an hour or more.

After a week of chasing the wild goose, we have isolated it to a single PR introduced in hyper 1.5.1:

https://github.com/hyperium/hyper/pull/3729

In these cases a tonic server:

            Server::builder()
                .add_service(GreeterServer::new(MyGreeter::default()))
                .serve_with_incoming_shutdown(incoming, async move {
                    shutdown_grpc_token.cancelled().await
                })
                .await
                .unwrap();

would never return and block for an hour or more after the shutdown_grpc_token CancellationToken was canceled.

We have a primary example non_upgrade_connections_can_block_shutdown and a secondary one we found along the way client_not_responding_example in this self contained example:

https://github.com/chamons/hyper_hung_client_example

With a patch.crates-io line, we can resolve this branch with the PR reverted and both of these example begin acting as expected.

Expected Behavior

It is not expected that passive tonic gRPC connections to a server that are unused would block shutdown. Before 1.5.1 they did not.

Actual Behavior

We block until the upstream instance restarts or otherwise drops all of its connections.

Additional Context

No response