Azure / Azure/azure-sdk-for-rust
eventhubs: two equal-epoch receivers trade a partition silently and both deliver events
- Dominant language
- Rust
- Stars
- 884
- Forks
- 365
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 111
Description
## Summary
Two receivers on one partition at the same owner level steal the link from each other about every 5 seconds. Neither reports an error. Both deliver every event, so a consumer sees duplicate processing with no signal.
`EventProcessor` opens every partition receiver at owner level 0, so two processor instances against one consumer group produce this state.
## Motivation
A live run measured the behavior on a real namespace, with the owner level reaching the wire. Two receivers on one partition, both at owner level 0, ran for 86 seconds. The broker accepted 15 attaches and refused none. It sent 13 Detach frames with condition `amqp:link:stolen` and the text `New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected.` Neither stream reported an error, and both printed every event.
The broker behaves correctly. At an equal epoch the later attacher wins, so it displaces the current holder and sends a stolen Detach. The loop is client-made. The displaced receiver reads an opaque link-state error, treats it as a link failure, re-attaches at the same epoch, and the broker accepts it. That attach displaces the other receiver, which does the same thing. The two receivers alternate.
`should_retry_receive_error` (`sdk/eventhubs/azure_messaging_eventhubs/src/common/recoverable/connection.rs:1026`) exists to stop this. It returns `ReturnError` for a stolen link, and its comment says a re-attach "would silently resurrect a displaced receiver". The guard does not fire here, because the condition is not readable on an in-flight receive with the published `azure_core_amqp` 1.1.0.
#4815 makes the condition readable, and with it the same contest reports `ConsumerDisconnected` and the loop stops. That is the right fix for the reporting. This issue is about the behavior underneath: a receiver that loses a link re-attaches into a steal-back loop, and nothing bounds it. The same shape returns for any condition the client cannot read, so the guard depends on error classification for a property that should not need it.
Duplicate delivery is the consumer-visible result. Both receivers read the same partition, so an application processes each event twice for as long as the contest runs.
## Proposal
Bound the re-attach loop, so a receiver cannot trade a link without limit. A count of re-attaches over a time window, with a terminal error when it passes the limit, gives a signal that does not depend on reading the AMQP condition.
Report a displacement that repeats. A receiver that attaches, loses the link, and re-attaches several times in a few seconds is in a contest, whatever the error kind says.
Consider whether `EventProcessor` should raise the owner level for a new claim instead of always using 0. A higher epoch makes the broker refuse the loser's re-attach, which ends the contest at the broker rather than in the client. Note that the epoch value is part of the observable contract, so this needs a decision rather than a patch.
Add a test for the contest. The unit tests cover the classification, and no test covers a receiver that re-attaches repeatedly.
Related to #4807 and #4815, which make the displacement reportable, and to #4439, which added the guard this issue shows is not sufficient.
Contributor guide
Research direction
Start with should_retry_receive_error in sdk/eventhubs/azure_messaging_eventhubs/src/common/recoverable/connection.rs:1026 and trace how EventProcessor opens partition receivers at owner level 0. Read the related behavior in #4815 and inspect the existing classification unit tests; no test currently covers repeated receiver re-attachments. Done should include an agreed contest-handling design, a regression test, and a terminal signal when the repeated displacement limit is exceeded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100