apache / apache/paimon

[Bug] Compact/Overwrite action may lose new added data

Open
#4,802 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.9

### Compute Engine

spark-3.2

### Minimal reproduce step

Spark SQL:
1. create bucket unaware table without primary keys like below:
create table paimon.paimon_test.test_compact(id int, data string) TBLPROPERTIES('bucket' = '-1');
2. Add few records into table paimon.paimon_test.test_compact;
insert into paimon.paimon_test.test_compact values(1, 'data01'), (2, 'data02'),(3, 'data03'),(4, 'data04');
3. call compact procedure:
CALL paimon.sys.compact(table => 'paimon_test.test_compact', order_strategy => 'order', order_by => 'id');
At the same time, add a new record with another spark-sql cli:
insert into paimon.paimon_test.test_compact values(666, 'data666')
4. Insert action is successful and data with values(666, 'data666') is missing after compact success.

### What doesn't meet your expectations?

The insert data with values(666, 'data666') should not be lost, It's better to throw a compact error instead of making data lose

### Anything else?

I checked the code, the compact action just implemented same as the overwrite action, and system always uses the latest snapshot to mark all the files as deleted instead of using the snapshot which the compact action invoked. Currently, the compact/overwrite action only obeys SNAPSHOT isolation instead of SERIALIZABLE isolation. I am not sure if it is as expected? however, current implement of compaction is very dangerous in our scene.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark SQL reproduction and the compact procedure, then inspect the compact and overwrite action paths that mark files deleted and handle snapshots. Trace how a concurrent insert is treated during compaction. Done means the new record is retained or compaction fails safely, with a regression test covering the concurrent operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.