HTTP2 multiplexed requests get stuck in presence of ... | hyperium/hyper

Nobody has claimed this yet.

Version
hyper 0.14.19
h2 0.3.13

Platform
Fedora 36

Linux fedora-wb-dev 5.17.11-300.fc36.x86_64 #1 SMP PREEMPT Wed May 25 15:04:05 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Description
We use hyper as a server and as a client in distributed application. Hyper client is configured as .http2_only(true).
Some GET requests are getting stuck. Those requests stream large response body (tens of megabytes). Some part of response body is transferred (several megabytes) and then both server and client are stuck with zero bytes being transferred. This only happens with HTTP2 in use.

The following code always reproduces this on my machine:

# Cargo.toml
[package]
name = "hyperreader"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
hyper = { version = "0.14.13", features = ["full"] }
tokio = { version = "1.12.0", features = ["full"] }
tokio-stream="*"

// main.rs
use std::convert::Infallible;
use std::error::Error;
use std::mem::forget;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use hyper::{Body, Request, Response};
use hyper::client::HttpConnector;
use hyper::service::{make_service_fn, service_fn};
use tokio::spawn;
use tokio::time::sleep;
use tokio_stream::StreamExt;

const URL: &'static str = "http://127.0.0.1:1234";
const PORT: u16 = 1234;
const BODY_SIZE: usize = 10 * 1024 * 1024;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    spawn(
        hyper::server::Server::bind(&SocketAddr::from(([127, 0, 0, 1], PORT)))
        .serve(make_service_fn(|_| async { Ok::<_, Infallible>(service_fn(serve_file)) }))
    );

    let client = hyper::client::Builder::default()
        .http2_only(true)
        .build::<_, Body>(HttpConnector::new());
    let client = Arc::new(client);

    for i in 0..3 {
        eprintln!("Sending {}", i);
        let uri = URL.parse().unwrap();
        let response = client.get(uri).await.unwrap();
        eprintln!("Received response {}", i);
        forget(response);
    }

    eprintln!("Sleeping 1 sec...");
    sleep(Duration::from_secs(1)).await;

    eprintln!("Sending request...");
    let uri = URL.parse().unwrap();
    let mut response = client.get(uri).await.unwrap();
    eprintln!("Received response...");
    let mut count = 0;
    while let Some(chunk) = response.body_mut().next().await {
        count += chunk?.len();
        println!("{}/{} bytes", count, BODY_SIZE);
    }

    Ok(())
}

async fn serve_file(_req: Request<Body>) -> Result<Response<Body>, Infallible> {
    let data = vec![0; BODY_SIZE];
    Ok(Response::new(Body::from(data)))
}

(note that the constants may be specific for a particular machine and may need tweaking for this to reproduce)

I expected to see this happen:
The program would run to completion and would print:

Sending 0
Received response 0
Sending 1
Received response 1
Sending 2
Received response 2
Sleeping 1 sec...
Sending request...
Received response...
// many lines skipped
10485760/10485760 bytes

Instead, this happened:

The program prints the following lines and then does nothing:

Sending 0
Received response 0
Sending 1
Received response 1
Sending 2
Received response 2
Sleeping 1 sec...
Sending request...
Received response...

In wireshark I see that neither client nor server don't send WINDOW_UPDATE frames during data transfer (except at the connection establishment). I suspect that this is a cause for this behavior.

Start by running the reproducer from Cargo.toml and main.rs, then observe the HTTP/2 traffic for WINDOW_UPDATE frames while the first responses are not consumed. Trace the client and server flow-control handling around the stalled requests. Done means the reproducer completes and the expected response body is fully read without hanging.

Written by the indexing model from the issue text.

HTTP2 multiplexed requests get stuck in presence of "slow readers", Stream/Connection window is not being updated

Description

Contributor guide

First steps

Research direction

Assessment