[Feature] Support Deletion Vector for Data Evolution Tables
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
In many AI workflows, users frequently need to delete data from DataEvolution tables after initial ingestion (e.g., removing low-quality samples post-feature engineering, deduplicating records, or excluding biased samples during iterative training).
The existing file-level Deletion Vector (DV) in AppendTable is incompatible with DataEvolution due to dynamic row-id semantics. Rewriting files to handle random deletions is also impractical because it:
* Amounts to a full-table rewrite (no better than INSERT OVERWRITE)
* Causes an explosion of small files
* Invalidates external indexes since row-ids must be reassigned before compaction
A tailored DV solution is therefore required for DataEvolution tables.
### Solution
We propose a range-based Deletion Vector approach that maps each RowId Range to a DV instead of one DV per DataFile. Key design points include:
* Introduce DeletionFileKey interface to unify FileName and RowIdRange keys, reusing existing deletion file logic
* Maintain schema compatibility by adding a new column rather than modifying existing ones
* Transparently apply DVs during DataEvolutionSplitRead with no read-path performance penalty
* Define clear strategies for Merge Into updates, Compaction (ignore vs. materialize DV), and Vector Index adaptation
### Anything else?
Please refer to [google docs](https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing) for full design.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked design document and the existing file-level Deletion Vector logic in AppendTable. Then trace DataEvolutionSplitRead and the stated Merge Into, Compaction, and Vector Index paths to understand the required range-based behavior. Done means implementing the proposed DeletionFileKey and schema-compatible approach plus the documented strategies, with validation for transparent DV application.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100