[Bug] delete from doesn't work with deletion-vector
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8
### Compute Engine
flink 1.7
### Minimal reproduce step
1、streaming write into a paimon table with deletion-vector enabled

2、choose a key in the table and run a batch statement to 'delete from' the paimon table
```
select * from paimon.test_fts.test_dvdown_1125 where k = '039f61ff8228b6515b88f0dcf1afaec41ccc3fd7614ff4763fa82dfd23455e127036eb152bc444fa34392dd42f92dbbccc67'
```
3、batch read from the paimon table with the key

- if we run the above select after 'delete from' and before streaming job taking a checkpoint : the output is empty
- if we run the above select after the checkpoint succeeding 'delete from': the key comes out again
### What doesn't meet your expectations?
The key is not deleted successfully.
We can also reproduce the weird issue with batch time travel:
With a snapshots history like below (snapshot 3 and 4 committed by 'delete from' , and others by streaming job)

- time travel read from snapshot 3 (which is append) gets the key
- time travel read from snapshot 4 (compact) or 5 (append) gets no output
- time travel read from snapshot 6 (compact) gets the key
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the minimal reproduction on Paimon 0.8 using Flink 1.7: enable deletion vectors, perform a streaming write, run the shown batch DELETE, and read the key before and after a checkpoint. Compare the results across the described snapshots and trace the deletion-vector handling for batch deletes and compaction. Done means the key stays deleted after the checkpoint and time-travel reads behave consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100