Allow Users to extend S3/GCS HoodieIncrSource to bring in additional columns from upstream
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
Current S3/GCS HoodieIncrSource reads file-paths from upstream tables and ingests to downstream tables. We need ability to extend this functionality by joining additional columns in the upstream table before writing to the downstream table.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7773
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the S3/GCS HoodieIncrSource entry point and tracing how upstream file paths are read before downstream writes. Identify the existing source configuration and tests around this flow. Done means an extension can join additional upstream columns and those columns are present in the downstream table without breaking current ingestion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100