Introduce Column Groups in Hudi to support wide tables (RFC-80)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
See RFC-80
https://github.com/apache/hudi/blob/master/rfc/rfc-80/rfc-80.md
**Why this feature is needed:**
Many streaming and ML use-cases employ wide tables to store 1000s of columns in a single table. Hudi storage must evolve to support these needs -- while keeping write amplification and table service efficiency
### User Experience
**How users will use this feature:**
Writers should be able to get automatic, intelligent splitting of columns across column groups. Table services must be able to change column groups within a file group. For advanced writers for e.g streaming jobs that specifically want to separate the columns they write to, users need to be able to control the mappings of columns to column groups for a given write operation.
- Configuration changes needed: WIP
- API changes: WIP
- Usage examples: WIP
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
https://github.com/apache/hudi/blob/master/rfc/rfc-80/rfc-80.md
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes
- Does this change storage format? Yes
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading RFC-80, which defines the proposed column-group storage changes. Trace the affected storage format, public APIs, configurations, and table-service behavior; the work is complete only when automatic splitting, table-service changes, and per-write column mappings are specified and implemented with usage guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100