apache / apache/hudi

RECORD_INDEX (RLI) silently drops records — index count < base data count, no error

Open
#19,225 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**

Writing to a COPY_ON_WRITE table with the Record Level Index enabled results in the record index having **fewer entries than the base data**, silently — no error or warning. The dropped records:
- are **absent from the RLI** (record index count < distinct base record keys),
- cause **silent read loss on RLI point-lookups** (a query by record key returns nothing though the row exists in the base files).

The dropped set is **different on every write** (zero overlap across runs — so it is not specific "bad" keys), and it correlates with record-key **entropy**: sequential keys drop 0; high-entropy keys drop the most. The count scales super-linearly with row count.

This reproduces with high-entropy record keys such as:
`concat('USL', substr(sha2(cast(id as string), 256), 1, 16))`

Observed shortfalls (same config/cluster, only the keys/size differ):

| record keys | N | shortfall |
|---|---|---|
| sequential (`SYN000…`) | 441M | 0 |
| high-entropy hex | 50M | 2,226 |
| high-entropy hex | 441M | 37,221 |

**What you expected:**

The record index should contain exactly one entry per distinct record key — RLI count == distinct base data record keys — with no silent drops.

**Steps to reproduce:**

1. Generate N unique high-entropy string keys and write to a COW table with RECORD_INDEX enabled:
```python
N = 50_000_000
synth = (spark.range(0, N)
.withColumn("id", expr("concat('USL', substr(sha2(cast(id as string), 256), 1, 16))"))
.withColumn("c", lit(1.0)).select("id", "c").dropDuplicates(["id"]))

opts = {
"hoodie.table.name": "rli_repro",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "id",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.NonpartitionedKeyGenerator",
"hoodie.metadata.enable": "true",
"hoodie.metadata.record.index.enable": "true",
"hoodie.index.type": "RECORD_INDEX",
}
synth.write.format("hudi").options(**opts).mode("overwrite").save(path)
```
2. Compare base-data distinct keys vs record-index entry count:
```python
data = spark.read.format("hudi").option("hoodie.metadata.enable", "false").load(path) \
.select("_hoodie_record_key").distinct().count()
rli = spark.read.format("hudi").load(path + "/.hoodie/metadata").where("type=5").count()
print(data, rli, data - rli) # -> 50000000 49997774 2226
```
3. data - rli is nonzero (2,226 here) — those keys are missing from the RLI. A point-lookup for a missing key (with data skipping enabled) returns 0 rows even though the record exists in the base files.

Full runnable script and raw output are attached (`rli_repro.py.txt`, `rli_repro_output.txt`).

### Environment

```markdown
**Hudi version:** 1.2.0 (also reproduced on 1.1.0)
**Query engine:** Spark 3.5
**Relevant configs:** COPY_ON_WRITE, hoodie.index.type=RECORD_INDEX, hoodie.metadata.record.index.enable=true, hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator, single writer, operation=upsert

### Logs and Stack Trace

No error or stack trace — the drop is silent. The only signal is the record-index count being lower than the distinct base-data record-key count. See attached rli_repro_output.txt (data=50000000, rli=49997774, shortfall=2226).

[rli_repro.py.txt](https://github.com/user-attachments/files/29812080/rli_repro.py.txt)

[rli_repro_output.txt](https://github.com/user-attachments/files/29812081/rli_repro_output.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the attached rli_repro.py.txt and rli_repro_output.txt, then reproduce the COPY_ON_WRITE write with RECORD_INDEX and compare distinct base keys against metadata entries of type 5. Trace the record-index write path that handles high-entropy keys and verify the point-lookup behavior. Done means every distinct base record key has one RLI entry, with no silent drops or missing lookups.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.