apache / apache/hudi

Introduce Column Groups in Hudi to support wide tables (RFC-80)

Open
#13,922 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**

See RFC-80

https://github.com/apache/hudi/blob/master/rfc/rfc-80/rfc-80.md

**Why this feature is needed:**

Many streaming and ML use-cases employ wide tables to store 1000s of columns in a single table. Hudi storage must evolve to support these needs -- while keeping write amplification and table service efficiency

### User Experience

**How users will use this feature:**

Writers should be able to get automatic, intelligent splitting of columns across column groups. Table services must be able to change column groups within a file group. For advanced writers for e.g streaming jobs that specifically want to separate the columns they write to, users need to be able to control the mappings of columns to column groups for a given write operation.

- Configuration changes needed: WIP
- API changes: WIP
- Usage examples: WIP

### Hudi RFC Requirements

**RFC PR link:** (if applicable)
https://github.com/apache/hudi/blob/master/rfc/rfc-80/rfc-80.md

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes
- Does this change storage format? Yes

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading RFC-80, which defines the proposed column-group storage changes. Trace the affected storage format, public APIs, configurations, and table-service behavior; the work is complete only when automatic splitting, table-service changes, and per-write column mappings are specified and implemented with usage guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.