Investigate whether meta fields could be omitted completely
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, even when meta fields are not populated, we still insert empty-string columns to adhere to the expected schema.
This has a non-trivial overhead of ~20% (relative to just writing dataset as is), since Spark had to essentially "re-write" the original row with prepended new fields.
We should investigate whether it's feasible to avoid adding empty-string columns completely if meta-fields are disabled.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4036
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is identified in the issue. Trace the Spark dataset-writing path that adds empty meta-field columns, then measure the disabled-meta-fields case against the current behavior; done means documenting whether omission is feasible and what compatibility constraints apply.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100