apache / apache/beam

Store&Read offset with KafkaIO

Open
#20,034 0 comments 0 reactions 0 assignees View on GitHub
io java kafka new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When creating a Pipeline through a KafkaIO object, I want to be able to specify the starting offset of consumption, and when traversing the message later, I can get the offset of the current message for storage in a relational database / NoSQL.
 
This feature is used to implement the exactly-once semantics of spark streaming consumption.
 
In the "Your own data store" section of the following url content, you can find how to achieve exactly-once semantics with spark streaming:
[http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html](http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html)

Imported from Jira [BEAM-8934](https://issues.apache.org/jira/browse/BEAM-8934). Original Jira may contain additional context.
Reported by: zjfplayer.

Contributor guide

Open the contributing guide

Research direction

Start with the KafkaIO entry point and compare the requested behavior with the linked Spark Streaming Kafka integration documentation. Define how a pipeline specifies its starting consumption offset and how traversal exposes the current message offset for external storage; done means both capabilities are available for exactly-once consumption.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.