apache / apache/hudi

Extend commit metadata with deduplication / precombine statistics (e.g. numDuplicates, numPrecombined)

Open
#18,976 1 comment 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Problem statement

Every Hudi write produces commit metadata that records per-file and per-partition write statistics — `numInserts`, `numUpdates`, `numWrites`, `numDeletes`, and related counters. These stats are the primary source of truth that operators, pipelines, and reconciliation tooling use to answer the question: *"How many records did my write actually produce?"*

However, when **deduplication** (`hoodie.combine.before.insert`) or **precombine** (during upsert) is enabled, multiple input records that share the same record key are collapsed into a single output record before anything is written. The commit metadata reports only the **final written count** — it does not report how many input records were collapsed along the way, or *why* the count shrank.

This creates an **observability gap**: a discrepancy between input record count and written record count cannot be attributed to a cause.

### Concrete example

Suppose an input RDD/Dataset contains 5 records that all share the same record key:

```
key=A, ts=1
key=A, ts=2
key=A, ts=3
key=A, ts=4
key=A, ts=5
```

With dedup/precombine enabled, Hudi keeps one record (say `ts=5`) and writes it. The commit metadata reports:

```
numInserts = 1
```

From this number alone, an operator **cannot tell the difference** between two very different scenarios:

1. **Expected behavior:** 4 records were legitimate duplicates, correctly collapsed by precombine. Data is fully intact. :white_check_mark:
2. **A bug / data loss:** records were silently dropped somewhere in the pipeline (a partitioning bug, a faulty merge, an index issue, etc.), and the "4 missing" records were *not* actually duplicates. :x:

Both scenarios look identical in commit metadata: `5 in -> 1 out`. There is no field that says "4 of these were dropped as duplicates."

### Why this matters

- **Data integrity / auditing:** Pipelines that reconcile source-vs-sink counts hit a dead end. A drop from 5 to 1 is unexplained, so it cannot be safely signed off as correct nor flagged as a real loss.
- **Debugging:** When a genuine data-loss bug occurs, there is no metadata signal distinguishing it from normal dedup behavior, making root-cause analysis much harder.
- **Trust:** Without dedup attribution, every count discrepancy requires manual, expensive investigation.

### Scope

This applies to **both** write paths:

- **Insert dedup** — duplicates dropped before insert when combine-before-insert is on.
- **Upsert precombine** — multiple incoming records for the same key combined down to one (and combined against the existing record on disk).

## Proposed solution

Extend Hudi commit metadata (`HoodieWriteStat` and the aggregated commit-level stats) with additional counters that make dedup/precombine explicit, for example:

- `numDuplicates` / `numRecordsDeduplicated` — input records dropped because they shared a key with another input record.
- `numPrecombined` — records eliminated by the precombine step specifically.

With these stats, the invariant becomes verifiable:

```
numInputRecords == numWrites + numDeletes + numDuplicates (+ numErrors)
```

When this equation balances, a count drop is provably explained by deduplication. When it does **not** balance, the gap points at a real bug — turning a silent ambiguity into an actionable signal.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.