apache / apache/iceberg

Scan planning fails permanently with "Can't index multiple DVs" when a snapshot has duplicate deletion vectors for one data file (read-side recovery)

Open
#17,206 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

main (development)

### Query engine

Spark

### Please describe the bug 🐞

**Summary:** Once a table ends up with more than one deletion vector (DV) referencing the same data file in a snapshot, **every scan of that table fails during planning** with:

```
Can't index multiple DVs for : and
```

thrown from `DeleteFileIndex.Builder.add(Map dvByPath, DeleteFile dv)` (`core/src/main/java/org/apache/iceberg/DeleteFileIndex.java`, ~L529-533). The table becomes effectively unreadable and there is no built-in way to recover it. This is core scan-planning code, so it affects every query engine, not just Spark.

**Why it throws:** The DV index is built by iterating all live delete-manifest entries in the snapshot (`DeleteFileIndex.Builder.build()` -> `loadDeleteFiles()` -> the loop at ~L501). For each DV, `add(...)` does `dvByPath.putIfAbsent(dv.referencedDataFile(), dv)` and throws a `ValidationException` if an entry already exists for that path. There is no deduplication or merge.

**Spec context:** The spec forbids more than one DV per data file per snapshot ("there can be at most one deletion vector for a given data file in a snapshot"; "Writers must ensure that there is at most one deletion vector per data file and must merge new deletes with existing vectors or position delete files"). So two DVs for one data file is always metadata corruption, and the `ValidationException` is a *correct* detection of an invalid state - but it leaves the table permanently unreadable.

**How a table reaches this state:** This is the read-side manifestation of a write-side conflict-detection gap (see #15966). When two concurrent row-level writers use a `conflictDetectionFilter` that is narrower than the set of data files they actually touch, `validateAddedDVs` can miss the conflict and allow two DVs for the same data file to be committed. #15966 fixes the *write* side and prevents future corruption, but it cannot heal tables that are **already** corrupted - those remain bricked.

**Steps to reproduce:** Produce a snapshot containing two live DV entries that reference the same data file (for example via the concurrent narrow-`conflictDetectionFilter` path described in #15966, or by crafting two delete manifests that each contain a DV for the same data file), then run any scan. Planning fails with `Can't index multiple DVs`.

**Expected behavior:** A reader should be able to recover from an already-corrupted table without data loss. Rather than throwing (and leaving the table unreadable), `DeleteFileIndex` should:

1. **Byte-identical duplicates** (same `location()`, `contentOffset()`, `contentSizeInBytes()`) - the same DV referenced from two manifest entries (e.g. a manifest-rewrite race): deduplicate silently, logging at DEBUG.
2. **Genuinely different DVs for the same data file**: **merge (union) their position bitmaps** by reading both Puffin blobs and combining them, analogous to the merge already performed on the write path, and log a WARNING about the spec violation. This preserves **all** deletes.
3. **Never silently pick one DV**, which would drop the other's deletes and resurrect deleted rows - a correctness violation worse than a scan failure.

**Proposed fix (I can contribute):** In `DeleteFileIndex.Builder.build()`, collect DVs per referenced data file, and in a post-pass deduplicate byte-identical DVs and union the position bitmaps of any genuinely different DVs (reusing the existing DV merge utility), logging a WARNING; index only the resulting merged DV. `FileIO` is available on the Builder. This is a read-side recovery/self-heal that complements the write-side fix in #15966. I would include a reproducer test.

### Willingness to contribute

- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in core/src/main/java/org/apache/iceberg/DeleteFileIndex.java, following Builder.build(), loadDeleteFiles(), and add() to understand how duplicate deletion vectors are indexed. Review the existing deletion-vector merge utility and issue #15966 for related conflict handling, then add the reproducer test described in the issue. Done means scans recover by deduplicating identical DVs or merging distinct ones without dropping deletes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.