Graceful shutdown never stops the server when there is an open sse connection

Version
hyper v0.14.17
axum v0.4.8

Platform
Linux generation 5.16.15-zen1-1-zen #1 ZEN SMP PREEMPT Thu, 17 Mar 2022 00:30:11 +0000 x86_64 GNU/Linux

Description
When the future passed to Server::with_graceful_shutdown completes, the server won't stop while there is an active sse connection.
I stumbled onto this using axum, so here is hwo to reproduce (essentially axum's sse example + the graceful shutdown example)

use std::{convert::Infallible, time::Duration};

use {
    axum::{
        response::sse::{Event, KeepAlive, Sse},
        routing::get,
        Router,
    },
    futures::stream::{self, Stream},
    tokio::signal,
    tokio_stream::StreamExt as _,
};

#[tokio::main]
async fn main() {
    let app = Router::new().route("/sse", get(sse_handler));

    axum::Server::bind(&"127.0.0.1:3300".parse().unwrap())
        .serve(app.into_make_service())
        .with_graceful_shutdown(shutdown_signal())
        .await
        .unwrap();
}

async fn sse_handler() -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    let stream = stream::repeat_with(|| Event::default().data("Hi!"))
        .map(Ok)
        .throttle(Duration::from_secs(1));

    Sse::new(stream).keep_alive(KeepAlive::default())
}

async fn shutdown_signal() {
    let ctrl_c = async {
        signal::ctrl_c().await.unwrap();
    };

    #[cfg(unix)]
    let terminate = async {
        signal::unix::signal(signal::unix::SignalKind::terminate())
            .expect("failed to install signal handler")
            .recv()
            .await;
    };

    #[cfg(not(unix))]
    let terminate = std::future::pending::<()>();

    tokio::select! {
        _ = ctrl_c => {},
        _ = terminate => {},
    }

    println!("Shutting down");
}

Run the server, connect with an sse client (I used curl -N localhost:3300/sse), try pressing Ctrl+c on the terminal running the server. Shutting down gets printed but the server doesn't shutdown until the sse client disconnects by it self. (new connections are not allowed, which is expected)

From my understanding the server keeps running until all connections finish, which is normally fine, but a little problematic when we are talking about sse, since an sse connection might not end at all.
Note that with websocket that issue doesn't exist, the server shuts down normally closing the ws connections.

Start with Server::with_graceful_shutdown and reproduce the behavior using the provided axum SSE example and an SSE client such as curl. Compare shutdown with an active SSE connection against the stated websocket behavior; done means graceful shutdown stops the server and handles the open SSE connection as expected.