[Bug] Delete data is not timely
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8
### Compute Engine
spark
### Minimal reproduce step
In our company's business, users will first delete data older than 30 days for a non-partitioned table, and then insert new data to update with the smaller sequence.field, however, the data will not be written because the deleted record's sequence.filed is more smaller. We must perform the full compaction after delete operation. And the full compaction is expensive for large data tables.
Is there any way to ensure that the delete operation is timely?
```sql
create table test_tb (
`req_id` STRING,
`ad_id` STRING,
`info` STRING,
`dt_seconds_asc` BIGINT
)USING paimon
TBLPROPERTIES(
'bucket' = '1',
'file.compression' = 'ZSTD',
'file.format' = 'PARQUET',
'primary-key' = 'req_id,ad_id',
'sequence.field' = 'dt_seconds_asc');
insert into test_tb values('a', 'b', 'info-1', 100);
delete from test_tb where dt_seconds_asc < 200;
insert into test_tb values('a', 'b', 'info-1', 50);
// audit log still -D rowkind, insert data '50' is useless
select * from `test_tb$audit_log`;
OK
rowkind req_id ad_id info dt_seconds_asc
-D a b info-1 100
// result is null
select * from test_tb;
```
### What doesn't meet your expectations?
Deleted data will not be included in the sequence.field comparison
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the SQL reproduction for the non-partitioned Paimon table using sequence.field and the DELETE followed by a lower-sequence INSERT. Trace how the delete appears in test_tb$audit_log and how the subsequent SELECT is evaluated; done means deleted rows no longer block later inserts with smaller sequence values without requiring full compaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100