hyperium / hyperium/h2

Logical connection deadlock occurs under high concurrency - outgoing data transmission stops

Open
#853 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.6k
Forks
382
Avg merge
20h 23m
Merged PRs (30d)
9

Description

Under load test scenarios, I observe that our client app enters a state where outgoing traffic stops flowing. Here is a PR with a test that reproduces the behavior: #852

The test is timing-dependent but reproduces the problem 100% reliably on my PC, just gets stuck at different points each time I run it.

Typical output:

Requests completed: 2 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000

thread 'logical_deadlock' panicked at tests\h2-tests\tests\deadlock.rs:117:17:
No requests completed in the last 2s, deadlock likely occurred

Increasing MAX_CONCURRENT_STREAMS beyond CONCURRENCY in the test file will make the test tend to pass, though I cannot say for sure that it does so always (insufficient data - the real app that was locking up did not exceed max_concurrent_streams as far as I know).

My best-effort investigation of the logs leading up to this suggests that when the timing aligns just right, we end up in a situation where flow control logic has allocated all connection capacity to requests that are in pending_open state but which never leave the pending_open state.

The leading theory is that this leads to a logical deadlock where:

  • None of the open streams have any capacity allocated to them, so their data is not being sent. They cannot get more capacity because the connection's send window is exhausted (conn=0).
  • All the streams that have assigned capacity are in pending_open state and are never leaving it (perhaps because max_concurrent_streams has been reached).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer in tests/h2-tests/tests/deadlock.rs and PR #852, then inspect the flow-control behavior around pending_open, max_concurrent_streams, and the exhausted connection send window. Done means the test no longer reaches the deadlock timeout and completes all 10000 requests under the reported concurrency conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.