Azure / Azure/azure-sdk-for-rust
Event Hubs: InMemoryCheckpointStore::claim_ownership aborts the whole batch on one ETag mismatch
- Dominant language
- Rust
- Stars
- 884
- Forks
- 365
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 109
Description
## Summary
`InMemoryCheckpointStore::claim_ownership` returns an error when one partition in the batch has an ETag mismatch. The caller gets no partitions at all, not even the partitions that the call already claimed. The production `BlobCheckpointStore` skips only the partition in conflict and returns the rest. The two stores do not agree on this behavior.
The work on #4594 and PR #4703 showed this gap. That PR only fixes the ETag rotation and the timestamp refresh on the renewal path. This gap is out of scope for that PR.
## Motivation
`claim_ownership` in `sdk/eventhubs/azure_messaging_eventhubs/src/in_memory_checkpoint_store.rs` (lines 102-112) loops over the requested ownerships. The loop calls `self.update_ownership(ownership)?` at line 106. `update_ownership` returns an error for an ETag mismatch at lines 67-79. The `?` operator propagates that error out of `claim_ownership`. One partition in conflict discards every partition that the loop claimed before it.
`claim_ownership` in `sdk/eventhubs/azure_messaging_eventhubs_checkpointstore_blob/src/checkpoint_store.rs` (lines 146-217) does not propagate the error. It maps a `PreconditionFailed` response or a `Conflict` response to `(None, None)` at lines 168-201. It writes a log record and continues the loop. It returns the partitions that it claimed.
An ETag mismatch is the normal outcome when two `EventProcessor` instances race for the same partition. It is expected traffic in a load-balanced consumer group, not an exceptional condition. The unit tests and the samples use the in-memory store. The difference hides bugs in the load balancer, or it makes a test fail for a condition that the blob store tolerates.
## Proposal
Make `InMemoryCheckpointStore::claim_ownership` agree with the blob store. The loop must catch the ETag-mismatch error from `update_ownership`. The loop must then write a log record with the partition ID and the ETag. The loop must then continue to the next ownership. The call must return the ownerships that it claimed.
Keep the error return for all other failures, for example an empty required field. An ETag mismatch is the only condition that the loop skips.
Add a test that claims a batch of partitions with one stale ETag. The test must make sure that the call succeeds. The test must also make sure that the call returns the other partitions.
Contributor guide
Assessment
This issue has not been assessed yet.