libp2p / libp2p/rust-yamux

Stream::poll_read hangs when the connection is closed

Open
#216 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
250
Forks
62
PR merge metrics
No merged PRs in 30d

Description

When a peer explicitly closes the connection using poll_close, consumers of Stream::poll_read never get notified of the state change, and keep waiting forever.

My use-case is a bit complex: I'm working on a peer-to-peer app that makes its best to obtain a QUIC connection with the remote peer. The connections are used to access gRPC services, and I've noticed that the h2 Connection task always gets stuck waiting on the yamux Stream after an explicit close when we managed to get a "better" connection.

I've tried to extract a minimal reproduction example.

Minimal reproduction example
// test-harness/tests/poll_close.rs

use futures::executor::LocalPool;
use futures::future;
use futures::prelude::*;
use futures::select;
use futures::task::Spawn;
use futures::task::SpawnExt;
use futures::AsyncReadExt;
use test_harness::*;
use yamux::{Config, Connection, Mode};

#[test]
fn poll_close_notifies_streams() {
    let _ = env_logger::try_init();
    let mut pool = LocalPool::new();

    // Create a connection pair using bounded endpoints
    let (server_endpoint, client_endpoint) = futures_ringbuf::Endpoint::pair(1024, 1024);

    // Create and spawn a "server" that echoes every message back to the client.
    let server = Connection::new(server_endpoint, Config::default(), Mode::Server);
    pool.spawner()
        .spawn_obj(
            async move { echo_server(server).await.unwrap() }
                .boxed()
                .into(),
        )
        .unwrap();

    // Create and spawn a "client" that sends messages expected to be echoed
    // by the server.
    let mut client = Connection::new(client_endpoint, Config::default(), Mode::Client);

    // Instanciate a stream on the client
    let stream = pool
        .run_until(future::poll_fn(|cx| client.poll_new_outbound(cx)))
        .unwrap();


    // Make the client connection progress
    let (tx_close, mut rx_close) = futures::channel::oneshot::channel();
    pool.spawner()
        .spawn_obj(
            async move {
                let mut should_close = false;

                loop {
                    let fut = if should_close {
                        future::poll_fn(|cx| client.poll_close(cx))
                            .map(|_| ())
                            .boxed()
                    } else {
                        future::poll_fn(|cx| client.poll_next_inbound(cx))
                            .map(|_| ())
                            .boxed()
                    };

                    select! {
                        _ = fut.fuse() => {
                            break;
                        }
                        _ = rx_close => {
                            should_close = true;
                        }
                    };
                }
            }
            .boxed()
            .into(),
        )
        .unwrap();

    let msg = vec![1u8; 42];

    // Send a message, then wait for a response that will never arrive since we'll explicitly close
    // the connection right after sending.
    pool.run_until(
        pool.spawner()
            .spawn_with_handle(
                async move {
                    let (mut reader, mut writer) = AsyncReadExt::split(stream);

                    writer.write_all(msg.as_ref()).await.unwrap();
                    tx_close.send(()).unwrap();

                    let mut buffer = vec![0; msg.len()];
                    // This should panic but instead waits forever
                    reader.read_exact(&mut buffer[..]).await.unwrap();

                    eprintln!("reading done");
                }
                .boxed(),
            )
            .unwrap(),
    );
}

From what I've gathered, both the server and the client respectively enter the Closing and Cleanup state when the client asks to close the connection. At this point, they both close the StreamCommand receivers and I believe this is the problem : the Stream::poll_read implementation did not subscribe to the sender and thus can never know it will never be woken up again.

Indeed, this task can only be woken up if the Connection uses the Wakers stored in the Shared state, but that only happens in Active::drop_all_streams. When the State transitions from Active to Closing or Cleanup the Shared state is lost so these Wakers can never be used again, and Stream::poll_read is stuck indefinitely.

I've tried two things so far:

  • duplicate drop_all_streams for Closing and calling it when poll_close returns Ready: this solves the problem in both the reproduction case above and in my app. I'm not sure this is semantically correct.
  • call sender.poll_ready in Stream::poll_read: this worked for the reproduction example, but not in my app

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction in test-harness/tests/poll_close.rs, then trace Stream::poll_read and the Connection::poll_close path through the Closing and Cleanup states. Run the reproduction and verify that the reader is woken and completes with the expected close-related error instead of waiting forever.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.