apache / apache/hudi

Clean up HoodieSparkSqlWriter

Open
#15,377 0 comments 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

HoodieSparkSqlWriter has evolved over the years w/ layer on layers of logic being stacked in it. It's reaching a point where it's virtually impossible to understand interactions and number of permutations b/w different features. One recent example is HUDI-4472.

We need to make sure that it's in a state where:
# Writing logic is easily traceable, and it's possible to understand how different features might impact the write-path.
# Make sure that it's structured in a way allowing plugging in new features/revisiting existing logic (for ex, allowing different path for row-writing not only for bulk-insert but other operations as well, etc)
#  

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4692
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1297

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating and reading HoodieSparkSqlWriter, then use HUDI-4472 and the linked HUDI-1297 epic to map the current write-path permutations. Done means the writer's logic is traceable and structured for the feature-path needs described in the issue; no specific test is named in the payload.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.