Store&Read offset with KafkaIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When creating a Pipeline through a KafkaIO object, I want to be able to specify the starting offset of consumption, and when traversing the message later, I can get the offset of the current message for storage in a relational database / NoSQL.
This feature is used to implement the exactly-once semantics of spark streaming consumption.
In the "Your own data store" section of the following url content, you can find how to achieve exactly-once semantics with spark streaming:
[http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html](http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html)
Imported from Jira [BEAM-8934](https://issues.apache.org/jira/browse/BEAM-8934). Original Jira may contain additional context.
Reported by: zjfplayer.
Contributor guide
Research direction
Start with the KafkaIO entry point and compare the requested behavior with the linked Spark Streaming Kafka integration documentation. Define how a pipeline specifies its starting consumption offset and how traversal exposes the current message offset for external storage; done means both capabilities are available for exactly-once consumption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100