[DISCUSSION] Deltastreamer - Reading commit checkpoint from Kafka instead of latest Hoodie commit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Our requirement is actually this: Supporting multiple deltastreamers writing to a single hudi table
[https://github.com/apache/hudi/issues/6718](https://github.com/apache/hudi/issues/6718)
[HUDI-5077](https://issues.apache.org/jira/browse/HUDI-5077)
We reviewed the deltastreamer code and noticed that the deltastreamer can read commits from Kafka consumer groups if it doesn't find the last checkpoint from Hoodie.
We're considering modifying the code to always read commits from Kafka, based on a new flag, by making changes in the Hudi deltastreamer.
Do you foresee any nuances or issues with this approach?
**Environment Description**
* Hudi version : 0.13.0
* Spark version : 3.3.2
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Hudi Deltastreamer checkpoint-handling entry point and review the related issues 6718 and HUDI-5077. Compare the existing fallback from Hoodie commits to Kafka consumer groups with the proposed always-read behavior, including the single-table, multiple-writer requirement. Done requires an agreed design and clarified implementation scope, not just a code change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, spark
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100