apache / apache/paimon

[Feature] Support Deletion Vector for Data Evolution Tables

Open
#8,322 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In many AI workflows, users frequently need to delete data from DataEvolution tables after initial ingestion (e.g., removing low-quality samples post-feature engineering, deduplicating records, or excluding biased samples during iterative training).

The existing file-level Deletion Vector (DV) in AppendTable is incompatible with DataEvolution due to dynamic row-id semantics. Rewriting files to handle random deletions is also impractical because it:

* Amounts to a full-table rewrite (no better than INSERT OVERWRITE)
* Causes an explosion of small files
* Invalidates external indexes since row-ids must be reassigned before compaction

A tailored DV solution is therefore required for DataEvolution tables.

### Solution

We propose a range-based Deletion Vector approach that maps each RowId Range to a DV instead of one DV per DataFile. Key design points include:

* Introduce DeletionFileKey interface to unify FileName and RowIdRange keys, reusing existing deletion file logic
* Maintain schema compatibility by adding a new column rather than modifying existing ones
* Transparently apply DVs during DataEvolutionSplitRead with no read-path performance penalty
* Define clear strategies for Merge Into updates, Compaction (ignore vs. materialize DV), and Vector Index adaptation

### Anything else?

Please refer to [google docs](https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing) for full design.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked design document and the existing file-level Deletion Vector logic in AppendTable. Then trace DataEvolutionSplitRead and the stated Merge Into, Compaction, and Vector Index paths to understand the required range-based behavior. Done means implementing the proposed DeletionFileKey and schema-compatible approach plus the documented strategies, with validation for transparent DV application.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.