Azure / Azure/azure-sdk-for-rust

Event Hubs: InMemoryCheckpointStore::claim_ownership aborts the whole batch on one ETag mismatch

Open
#4,867 0 comments 1 reaction 1 assignee Claimed by @j7nw4r View on GitHub
Client Event Hubs
Dominant language
Rust
Stars
884
Forks
365
Avg merge
2d 19h
Merged PRs (30d)
109

Description

## Summary

`InMemoryCheckpointStore::claim_ownership` returns an error when one partition in the batch has an ETag mismatch. The caller gets no partitions at all, not even the partitions that the call already claimed. The production `BlobCheckpointStore` skips only the partition in conflict and returns the rest. The two stores do not agree on this behavior.

The work on #4594 and PR #4703 showed this gap. That PR only fixes the ETag rotation and the timestamp refresh on the renewal path. This gap is out of scope for that PR.

## Motivation

`claim_ownership` in `sdk/eventhubs/azure_messaging_eventhubs/src/in_memory_checkpoint_store.rs` (lines 102-112) loops over the requested ownerships. The loop calls `self.update_ownership(ownership)?` at line 106. `update_ownership` returns an error for an ETag mismatch at lines 67-79. The `?` operator propagates that error out of `claim_ownership`. One partition in conflict discards every partition that the loop claimed before it.

`claim_ownership` in `sdk/eventhubs/azure_messaging_eventhubs_checkpointstore_blob/src/checkpoint_store.rs` (lines 146-217) does not propagate the error. It maps a `PreconditionFailed` response or a `Conflict` response to `(None, None)` at lines 168-201. It writes a log record and continues the loop. It returns the partitions that it claimed.

An ETag mismatch is the normal outcome when two `EventProcessor` instances race for the same partition. It is expected traffic in a load-balanced consumer group, not an exceptional condition. The unit tests and the samples use the in-memory store. The difference hides bugs in the load balancer, or it makes a test fail for a condition that the blob store tolerates.

## Proposal

Make `InMemoryCheckpointStore::claim_ownership` agree with the blob store. The loop must catch the ETag-mismatch error from `update_ownership`. The loop must then write a log record with the partition ID and the ETag. The loop must then continue to the next ownership. The call must return the ownerships that it claimed.

Keep the error return for all other failures, for example an empty required field. An ETag mismatch is the only condition that the loop skips.

Add a test that claims a batch of partitions with one stale ETag. The test must make sure that the call succeeds. The test must also make sure that the call returns the other partitions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.