Logical connection deadlock occurs under high concurrency - outgoing data transmission stops
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 382
- Avg merge
- 20h 23m
- Merged PRs (30d)
- 9
Description
Under load test scenarios, I observe that our client app enters a state where outgoing traffic stops flowing. Here is a PR with a test that reproduces the behavior: #852
The test is timing-dependent but reproduces the problem 100% reliably on my PC, just gets stuck at different points each time I run it.
Typical output:
Requests completed: 2 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
Requests completed: 32 of 10000
thread 'logical_deadlock' panicked at tests\h2-tests\tests\deadlock.rs:117:17:
No requests completed in the last 2s, deadlock likely occurred
Increasing MAX_CONCURRENT_STREAMS beyond CONCURRENCY in the test file will make the test tend to pass, though I cannot say for sure that it does so always (insufficient data - the real app that was locking up did not exceed max_concurrent_streams as far as I know).
My best-effort investigation of the logs leading up to this suggests that when the timing aligns just right, we end up in a situation where flow control logic has allocated all connection capacity to requests that are in pending_open state but which never leave the pending_open state.
The leading theory is that this leads to a logical deadlock where:
- None of the open streams have any capacity allocated to them, so their data is not being sent. They cannot get more capacity because the connection's send window is exhausted (
conn=0). - All the streams that have assigned capacity are in
pending_openstate and are never leaving it (perhaps becausemax_concurrent_streamshas been reached).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproducer in tests/h2-tests/tests/deadlock.rs and PR #852, then inspect the flow-control behavior around pending_open, max_concurrent_streams, and the exhausted connection send window. Done means the test no longer reaches the deadlock timeout and completes all 10000 requests under the reported concurrency conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100