Refactor DeltaStreamer config to use HoodieIngestionConfig
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
This is a follow up to HUDI-5872
HoodieIngestionService should be constructed with HoodieIngestionConfig as the standard config class to be able to work with DeltaStreamer's Config and take in spark context and hadoop conf, along with other props.
This will allow fully customizable HoodeIngestionService to work with HoodieDeltaStreamer
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5874
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how HoodieIngestionService is constructed for HoodieDeltaStreamer and review the related HUDI-5872 follow-up and HUDI-5874 context. The work is done when the service uses HoodieIngestionConfig as its standard configuration and accepts the Spark context, Hadoop configuration, and other properties described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100