[FEATURE] Hybrid CoW/MoR: write base files directly for "hot" file groups during ingestion
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
## What the feature achieves
Decide **per file group, at ingestion time**, whether to append a delta log file or write a new base file directly — turning the table into a **per-file-group hybrid of Copy-on-Write and Merge-on-Read**:
- **Cold file groups** (sparse updates, e.g. older partitions) → **log files** (normal MoR).
- **Hot file groups** (heavy updates, e.g. recent partitions) → **write a new base file directly**, skipping the log-then-immediately-compact round trip.
The routing decision is **workload-aware**: it reuses the per-file-group statistics Hudi already computes during workload profiling (insert/update counts, current file group size, estimated batch size), so no extra scan is required to make the call.
**Benefits:**
- No separate compaction step for hot file groups.
- Data is materialized into a base file **at delta-commit time** → freshness preserved, no compaction-lag window, no async compaction backlog.
- Write amplification is only paid where it's unavoidable (the genuinely hot groups); cold groups stay cheap and log-based.
- Cheaper and more consistent snapshot reads — hot groups have a fresh base file with few/no logs to merge, cold groups merge only small logs.
## Why this feature is needed
Our critical datasets are **Copy-on-Write**, so even tiny updates rewrite the *entire* base file for a file group → heavy **write amplification**.
Moving to **Merge-on-Read** would fix the write cost, but introduces new problems for this workload:
- **Snapshot reads merge base + log on the fly** → slower, costlier queries.
- **Freshness is the top priority** — these tables feed many downstream datasets, so we can't tolerate a staleness window waiting on compaction.
**Key pattern:** updates are highly skewed — a small set of file groups in recent partitions take most of the updates, while older partitions barely change. This skew is exactly what makes a per-file-group decision worthwhile.
### Options we ruled out
| Approach | Why it fails |
|---|---|
| MoR + Read-Optimized view + aggressive compaction | Compaction rewrites base files so often we're back to CoW-level write amplification. |
| MoR + compact only hot file groups + Read-Optimized view | RO view becomes **inconsistent**: uncompacted file groups silently drop their recent updates. |
| MoR + compact only hot file groups + **snapshot view** | Works, but still needs a **separate compaction step** right after ingestion, which can delay freshness further — queries that start before compaction finishes take the merge hit. |
The proposed feature removes that residual compaction step entirely by materializing the
hot-group updates as base files during the delta commit itself.
### User Experience
**How users will use this feature:**
Users **opt in** on a per-table basis to **workload-aware updates** on a MoR table. Once enabled, ingestion automatically routes heavy-update ("hot") file groups to a direct base-file write while leaving cold file groups as normal log appends — no manual per-partition compaction tuning required. Existing readers continue to use the snapshot/real-time view and simply see fresher, cheaper-to-read data for the hot groups.
**Configuration changes needed:** _(work in progress)_
- An opt-in toggle to enable workload-aware updates on a MoR table.
- A "hot file group" threshold (e.g. estimated log size vs. base file size, or update
count/ratio per file group).
- An optional cap on base files rewritten per commit, to bound ingestion latency.
**API changes:** _(work in progress)_
**Usage examples:** _(work in progress)_
### Hudi RFC Requirements
**RFC PR link:** (N/A for now)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs?**Yes** — adds new (opt-in) write configuration properties; no breaking changes to existing APIs.
- Does this change storage format? **No new file formats** — base and log files remain standard.
Contributor guide
No contributing guide indexed for this repository
Research direction
No implementation files, tests, or entry points are named in the issue. Start by tracing Hudi's MoR ingestion path and the existing workload-profiling statistics, then determine where per-file-group routing and opt-in configuration belong; done requires settled threshold, cap, and API decisions plus coverage for hot and cold groups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100