RECORD_INDEX (RLI) silently drops records — index count < base data count, no error
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Writing to a COPY_ON_WRITE table with the Record Level Index enabled results in the record index having **fewer entries than the base data**, silently — no error or warning. The dropped records:
- are **absent from the RLI** (record index count < distinct base record keys),
- cause **silent read loss on RLI point-lookups** (a query by record key returns nothing though the row exists in the base files).
The dropped set is **different on every write** (zero overlap across runs — so it is not specific "bad" keys), and it correlates with record-key **entropy**: sequential keys drop 0; high-entropy keys drop the most. The count scales super-linearly with row count.
This reproduces with high-entropy record keys such as:
`concat('USL', substr(sha2(cast(id as string), 256), 1, 16))`
Observed shortfalls (same config/cluster, only the keys/size differ):
| record keys | N | shortfall |
|---|---|---|
| sequential (`SYN000…`) | 441M | 0 |
| high-entropy hex | 50M | 2,226 |
| high-entropy hex | 441M | 37,221 |
**What you expected:**
The record index should contain exactly one entry per distinct record key — RLI count == distinct base data record keys — with no silent drops.
**Steps to reproduce:**
1. Generate N unique high-entropy string keys and write to a COW table with RECORD_INDEX enabled:
```python
N = 50_000_000
synth = (spark.range(0, N)
.withColumn("id", expr("concat('USL', substr(sha2(cast(id as string), 256), 1, 16))"))
.withColumn("c", lit(1.0)).select("id", "c").dropDuplicates(["id"]))
opts = {
"hoodie.table.name": "rli_repro",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "id",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.NonpartitionedKeyGenerator",
"hoodie.metadata.enable": "true",
"hoodie.metadata.record.index.enable": "true",
"hoodie.index.type": "RECORD_INDEX",
}
synth.write.format("hudi").options(**opts).mode("overwrite").save(path)
```
2. Compare base-data distinct keys vs record-index entry count:
```python
data = spark.read.format("hudi").option("hoodie.metadata.enable", "false").load(path) \
.select("_hoodie_record_key").distinct().count()
rli = spark.read.format("hudi").load(path + "/.hoodie/metadata").where("type=5").count()
print(data, rli, data - rli) # -> 50000000 49997774 2226
```
3. data - rli is nonzero (2,226 here) — those keys are missing from the RLI. A point-lookup for a missing key (with data skipping enabled) returns 0 rows even though the record exists in the base files.
Full runnable script and raw output are attached (`rli_repro.py.txt`, `rli_repro_output.txt`).
### Environment
```markdown
**Hudi version:** 1.2.0 (also reproduced on 1.1.0)
**Query engine:** Spark 3.5
**Relevant configs:** COPY_ON_WRITE, hoodie.index.type=RECORD_INDEX, hoodie.metadata.record.index.enable=true, hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator, single writer, operation=upsert
### Logs and Stack Trace
No error or stack trace — the drop is silent. The only signal is the record-index count being lower than the distinct base-data record-key count. See attached rli_repro_output.txt (data=50000000, rli=49997774, shortfall=2226).
[rli_repro.py.txt](https://github.com/user-attachments/files/29812080/rli_repro.py.txt)
[rli_repro_output.txt](https://github.com/user-attachments/files/29812081/rli_repro_output.txt)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the attached rli_repro.py.txt and rli_repro_output.txt, then reproduce the COPY_ON_WRITE write with RECORD_INDEX and compare distinct base keys against metadata entries of type 5. Trace the record-index write path that handles high-entropy keys and verify the point-lookup behavior. Done means every distinct base record key has one RLI entry, with no silent drops or missing lookups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100