apache / apache/paimon

[Feature] Support deletion vectors for chain table

Open
#8,573 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Currently, chain tables support merge-on-read across snapshot and delta branches, but they do not support deletion vectors. For workloads that use `deletion-vectors.enabled=true` to speed up point deletes and updates, users cannot benefit from chain tables.

### Solution
Expose DV-marked rows as -D tombstone KeyValues during chain read, so the merge function can apply cross-branch deletes correctly. The normal (non-chain) read path still uses `ApplyDeletionVectorReader` for branch-local filtering.

- ChainSplit: carry deletion files from both snapshot and delta branches alongside their data files.
- ChainGroupReadTable: collect the relevant deletion files for each data file when constructing ChainSplit.
- ExposeDeletionKeyValueReader (new): wraps a `KeyValueDataFileRecordReader` and flips valueKind to `RowKind.DELETE` for positions marked by the deletion vector, so the merge function receives the delete record instead of silently dropping the row.
- ChainKeyValueFileReaderFactory: use `ExposeDeletionKeyValueReader` when a non-empty DV is present.
- MergeFileSplitRead#createChainReader: unwrap `LookupMergeFunction.Factory` so cross-branch records are merged by the inner merge function (e.g., DeduplicateMergeFunction) using the sequence field.
- Validation: restrict chain-table DV support to the DEDUPLICATE merge engine.

### Anything else?
#### Why only DEDUPLICATE?
`DEDUPLICATE` and `PARTIAL_UPDATE` handle `-U` / `-D` differently. In `DEDUPLICATE` both `-U` and `-D` mean retract/delete, so a DV tombstone maps cleanly. In `PARTIAL_UPDATE`, `-U` is a branch-local retraction that must not cross-delete snapshot records, while -D is a real delete. Because DV bitmap marks are exposed uniformly as -D tombstones during chain read, the `-U`/`-D` distinction is lost, which would cause delta -U operations to incorrectly delete matching snapshot records. Therefore DV is only enabled for `DEDUPLICATE` in this change.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read ChainSplit, ChainGroupReadTable, ChainKeyValueFileReaderFactory, and MergeFileSplitRead#createChainReader first to trace how chain reads assemble branch files and merge records. Done means deletion vectors produce -D tombstone KeyValues across snapshot and delta branches, support is restricted to DEDUPLICATE, and the normal non-chain path remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.