[Feature] Support deletion vectors for chain table
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Currently, chain tables support merge-on-read across snapshot and delta branches, but they do not support deletion vectors. For workloads that use `deletion-vectors.enabled=true` to speed up point deletes and updates, users cannot benefit from chain tables.
### Solution
Expose DV-marked rows as -D tombstone KeyValues during chain read, so the merge function can apply cross-branch deletes correctly. The normal (non-chain) read path still uses `ApplyDeletionVectorReader` for branch-local filtering.
- ChainSplit: carry deletion files from both snapshot and delta branches alongside their data files.
- ChainGroupReadTable: collect the relevant deletion files for each data file when constructing ChainSplit.
- ExposeDeletionKeyValueReader (new): wraps a `KeyValueDataFileRecordReader` and flips valueKind to `RowKind.DELETE` for positions marked by the deletion vector, so the merge function receives the delete record instead of silently dropping the row.
- ChainKeyValueFileReaderFactory: use `ExposeDeletionKeyValueReader` when a non-empty DV is present.
- MergeFileSplitRead#createChainReader: unwrap `LookupMergeFunction.Factory` so cross-branch records are merged by the inner merge function (e.g., DeduplicateMergeFunction) using the sequence field.
- Validation: restrict chain-table DV support to the DEDUPLICATE merge engine.
### Anything else?
#### Why only DEDUPLICATE?
`DEDUPLICATE` and `PARTIAL_UPDATE` handle `-U` / `-D` differently. In `DEDUPLICATE` both `-U` and `-D` mean retract/delete, so a DV tombstone maps cleanly. In `PARTIAL_UPDATE`, `-U` is a branch-local retraction that must not cross-delete snapshot records, while -D is a real delete. Because DV bitmap marks are exposed uniformly as -D tombstones during chain read, the `-U`/`-D` distinction is lost, which would cause delta -U operations to incorrectly delete matching snapshot records. Therefore DV is only enabled for `DEDUPLICATE` in this change.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Read ChainSplit, ChainGroupReadTable, ChainKeyValueFileReaderFactory, and MergeFileSplitRead#createChainReader first to trace how chain reads assemble branch files and merge records. Done means deletion vectors produce -D tombstone KeyValues across snapshot and delta branches, support is restricted to DEDUPLICATE, and the normal non-chain path remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100