apache / apache/paimon

[Bug] delete from doesn't work with deletion-vector

Open
#4,588 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8

### Compute Engine

flink 1.7

### Minimal reproduce step

1、streaming write into a paimon table with deletion-vector enabled
![image](https://github.com/user-attachments/assets/76faf6ff-271b-41c8-b5b7-6b5f81c8704e)

2、choose a key in the table and run a batch statement to 'delete from' the paimon table
```
select * from paimon.test_fts.test_dvdown_1125 where k = '039f61ff8228b6515b88f0dcf1afaec41ccc3fd7614ff4763fa82dfd23455e127036eb152bc444fa34392dd42f92dbbccc67'
```

3、batch read from the paimon table with the key
![image](https://github.com/user-attachments/assets/4a2ae11b-391f-40ad-a56c-a06e18dda05f)
- if we run the above select after 'delete from' and before streaming job taking a checkpoint : the output is empty
- if we run the above select after the checkpoint succeeding 'delete from': the key comes out again

### What doesn't meet your expectations?

The key is not deleted successfully.
We can also reproduce the weird issue with batch time travel:
With a snapshots history like below (snapshot 3 and 4 committed by 'delete from' , and others by streaming job)
![image](https://github.com/user-attachments/assets/c7221fb6-f81b-4859-bdd4-0519f76add7c)
- time travel read from snapshot 3 (which is append) gets the key
- time travel read from snapshot 4 (compact) or 5 (append) gets no output
- time travel read from snapshot 6 (compact) gets the key

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal reproduction on Paimon 0.8 using Flink 1.7: enable deletion vectors, perform a streaming write, run the shown batch DELETE, and read the key before and after a checkpoint. Compare the results across the described snapshots and trace the deletion-vector handling for batch deletes and compaction. Done means the key stays deleted after the checkpoint and time-travel reads behave consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.