Consolidate S3EventsHoodieIncrSourceConfig into HoodieIncrSourceConfig
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
should deprecate repeated configs in S3EventsHoodieIncrSourceConfig and make people use keys in HoodieIncrSourceConfig and CloudSourceConfig going forward
Config key like org.apache.hudi.utilities.config.CloudSourceConfig#SPARK_DATASOURCE_OPTIONS should also be renamed to something like `hoodie.deltastreamer.source.cloud_incr.spark.datasource.options`
previous discussion
https://github.com/apache/hudi/pull/8184#discussion_r1148455190
https://github.com/apache/hudi/pull/8184#discussion_r1148454968
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6020
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading S3EventsHoodieIncrSourceConfig, HoodieIncrSourceConfig, and CloudSourceConfig, then review the linked discussion for the intended consolidation. Trace the repeated configuration keys and their uses, including SPARK_DATASOURCE_OPTIONS; the work is done when the old entries are deprecated, callers use the shared configs, and the cloud incremental key naming is updated consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100