Clean up HoodieSparkSqlWriter
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
HoodieSparkSqlWriter has evolved over the years w/ layer on layers of logic being stacked in it. It's reaching a point where it's virtually impossible to understand interactions and number of permutations b/w different features. One recent example is HUDI-4472.
We need to make sure that it's in a state where:
# Writing logic is easily traceable, and it's possible to understand how different features might impact the write-path.
# Make sure that it's structured in a way allowing plugging in new features/revisiting existing logic (for ex, allowing different path for row-writing not only for bulk-insert but other operations as well, etc)
#
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4692
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1297
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating and reading HoodieSparkSqlWriter, then use HUDI-4472 and the linked HUDI-1297 epic to map the current write-path permutations. Done means the writer's logic is traceable and structured for the feature-path needs described in the issue; no specific test is named in the payload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100