confluentinc / confluentinc/parallel-consumer
Handling Partition Revocation in Parallel-Consumer Leading to Duplicate Event Processing
- Dominant language
- Java
- Stars
- 299
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
During the investigation of the parallel-consumer with reactor, we observed the following behaviour:
1. The consumer starts processing an event from a partition.
2. While the event is being processed, the partition gets revoked, and the current progress (highest sequential offset & metadata with out of order offsets) gets committed.
3. The processing of the event continues and eventually completes. However, when attempting to update the state for the partition, it is discovered that the partition is now in a `RemovedPartitionState` due to the revocation. As a result, the state update is ignored.
4. Subsequently, the same partition is reassigned to the same consumer. The consumer reads the latest state from the committed offset, but since the previous state update was ignored (step 3), the state is stale and does not reflect the processing that was completed.
Related Logs: https://gist.github.com/ajax-levashov-m/29efe07beff585d659ba4d562622721f
Due to this behaviour, event processing is performed twice. The first processing attempt is effectively discarded because the partition state update is ignored after the revocation. When the partition is reassigned, the consumer starts processing from the previously committed offset, resulting in the event being processed again.
Question:
Are there any ways to mitigate this behaviour and use parallel-consumer for non-idempotent workloads? It would be beneficial to have a mechanism for the consumer to complete the processing of in-flight messages within a defined timeout before revoking the partition, or at least cancel the current processing when it involves reactor processor.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.