dapr / dapr/components-contrib
support kafka EOS (exactly only once semantics)
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Describe the feature
Support **Kafka exactly-once semantics (EOS)** in the Dapr Kafka pub/sub component so that applications can get exactly-once processing when enabled via component configuration. Users would be able to opt in (e.g. via metadata) and have the component ensure that each message's offset commit and downstream publish succeed or fail atomically within a Kafka transaction, so duplicate processing is avoided in all scenarios — including failures and retries.
Kafka has supported exactly-once semantics since 0.11 (transactional produce/consume and offset commit). (see e.g. [Plans to support upcoming Kafka v0.11 ? (#901)](https://github.com/IBM/sarama/issues/901) and [Exactly Once arrived with kafka 0.11 ? (#977)](https://github.com/IBM/sarama/issues/977)); the feature request is to expose and certify this capability in the Dapr component. At a high level, enabling EOS would mean:
- producing within a transaction (so sends are atomic),
- consuming with read-committed isolation
- committing consumer offsets in the same transaction so processing and offset commit succeed or fail together—avoiding duplicates and double-processing when the component is used as configured.
**Business value:** Applications that need exactly-once or no-duplicate processing over Kafka could rely on Dapr’s Kafka component instead of building their own transaction handling, which is especially useful for event-driven workflows, audit trails, and pipelines that must not double-process messages.
## Release Note
RELEASE NOTE: **ADD** Kafka pub/sub component supports exactly-once semantics (EOS) via optional metadata, enabling transactional produce and consume so applications can avoid duplicate processing.
Contributor guide
Assessment
This issue has not been assessed yet.