apache / apache/hudi

hoodie.metadata.compact.max.delta.commits behaving unexpectedly

Open
#18,041 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
with hoodie.metadata.compact.max.delta.commits set to 1, the table's metadata table is only compacted every 3 delta commits

**What you expected:**
compaction after every delta commit

**Steps to reproduce:**
1. It should suffice to just set up a COW table and provide some upserts to it with hoodie.metadata.compact.max.delta.commits set to 1, then check the timeline of the metadata table. If not, I can dig in more.

### Environment

**Hudi version:** 1.1.1
**Query engine:** Spark 4.0.1 (via Spark Connect)
**Relevant configs:**
hoodie.metadata.index.column.stats.enable: true,
hoodie.metadata.enable: true,
hoodie.metadata.compact.max.delta.commits: 1,
hoodie.datasource.write.operation: upsert,
hoodie.datasource.write.table.type: COPY_ON_WRITE

### Logs and Stack Trace

Image

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with a COPY_ON_WRITE table, upserts, and hoodie.metadata.compact.max.delta.commits set to 1 in Spark 4.0.1. Inspect the metadata table timeline after each delta commit and compare the observed compaction interval with the expected one-commit interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.