apache / apache/paimon

[Bug] Delete data is not timely

Open
#4,767 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8

### Compute Engine

spark

### Minimal reproduce step

In our company's business, users will first delete data older than 30 days for a non-partitioned table, and then insert new data to update with the smaller sequence.field, however, the data will not be written because the deleted record's sequence.filed is more smaller. We must perform the full compaction after delete operation. And the full compaction is expensive for large data tables.

Is there any way to ensure that the delete operation is timely?

```sql
create table test_tb (
`req_id` STRING,
`ad_id` STRING,
`info` STRING,
`dt_seconds_asc` BIGINT
)USING paimon
TBLPROPERTIES(
'bucket' = '1',
'file.compression' = 'ZSTD',
'file.format' = 'PARQUET',
'primary-key' = 'req_id,ad_id',
'sequence.field' = 'dt_seconds_asc');

insert into test_tb values('a', 'b', 'info-1', 100);

delete from test_tb where dt_seconds_asc < 200;

insert into test_tb values('a', 'b', 'info-1', 50);

// audit log still -D rowkind, insert data '50' is useless
select * from `test_tb$audit_log`;
OK
rowkind req_id ad_id info dt_seconds_asc
-D a b info-1 100

// result is null
select * from test_tb;

```

### What doesn't meet your expectations?

Deleted data will not be included in the sequence.field comparison

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the SQL reproduction for the non-partitioned Paimon table using sequence.field and the DELETE followed by a lower-sequence INSERT. Trace how the delete appears in test_tb$audit_log and how the subsequent SELECT is evaluated; done means deleted rows no longer block later inserts with smaller sequence values without requiring full compaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.