dapr / dapr/components-contrib

Ability to "ResetOffset" for Kafka pub/sub

Open
#1,627 9 comments 0 reactions 0 assignees View on GitHub
area/runtime/pubsub discussion good first issue size/M
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## In what area(s)?

/area components (kafka pub/sub)

## Describe the feature
The current kafka pub/sub component targets just the minimum interfaces to publish and subscribe. We need an enhancement to the kafka component to support "ResetOffets". This means, we need to be able to invoke an API call to the dapr sidecar to "reset" the current offset to a previous value so that messages can be replayed.
As a side note, when dapr calls our services on an output binding for delivering messages, include the "offset" number so that clients can keep track of Message-Id to offset mapping.

**UPDATE**

The solution would involve handling metadata from PubSub subscriber's response and let each component implement custom handling of metadata. So, the subscriber can respond with a custom metadata to enable these scenarios.

Contributor guide

Open the contributing guide

Research direction

Start by examining the Kafka pub/sub component and the Dapr sidecar pub/sub API described in the issue, then trace how subscriber responses currently carry metadata. Clarify the component-specific metadata contract and how an API call would select a prior offset. Done means the design is agreed and both offset replay and delivery of offset metadata are supported and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.