dapr / dapr/components-contrib

support kafka EOS (exactly only once semantics)

Open
#4,301 3 comments 0 reactions 1 assignee Claimed by @javier-aliaga View on GitHub
kind/enhancement
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Describe the feature
Support **Kafka exactly-once semantics (EOS)** in the Dapr Kafka pub/sub component so that applications can get exactly-once processing when enabled via component configuration. Users would be able to opt in (e.g. via metadata) and have the component ensure that each message's offset commit and downstream publish succeed or fail atomically within a Kafka transaction, so duplicate processing is avoided in all scenarios — including failures and retries.

Kafka has supported exactly-once semantics since 0.11 (transactional produce/consume and offset commit). (see e.g. [Plans to support upcoming Kafka v0.11 ? (#901)](https://github.com/IBM/sarama/issues/901) and [Exactly Once arrived with kafka 0.11 ? (#977)](https://github.com/IBM/sarama/issues/977)); the feature request is to expose and certify this capability in the Dapr component. At a high level, enabling EOS would mean:

- producing within a transaction (so sends are atomic),
- consuming with read-committed isolation
- committing consumer offsets in the same transaction so processing and offset commit succeed or fail together—avoiding duplicates and double-processing when the component is used as configured.

**Business value:** Applications that need exactly-once or no-duplicate processing over Kafka could rely on Dapr’s Kafka component instead of building their own transaction handling, which is especially useful for event-driven workflows, audit trails, and pipelines that must not double-process messages.

## Release Note

RELEASE NOTE: **ADD** Kafka pub/sub component supports exactly-once semantics (EOS) via optional metadata, enabling transactional produce and consume so applications can avoid duplicate processing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.