[Bug] Compact/Overwrite action may lose new added data
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.9
### Compute Engine
spark-3.2
### Minimal reproduce step
Spark SQL:
1. create bucket unaware table without primary keys like below:
create table paimon.paimon_test.test_compact(id int, data string) TBLPROPERTIES('bucket' = '-1');
2. Add few records into table paimon.paimon_test.test_compact;
insert into paimon.paimon_test.test_compact values(1, 'data01'), (2, 'data02'),(3, 'data03'),(4, 'data04');
3. call compact procedure:
CALL paimon.sys.compact(table => 'paimon_test.test_compact', order_strategy => 'order', order_by => 'id');
At the same time, add a new record with another spark-sql cli:
insert into paimon.paimon_test.test_compact values(666, 'data666')
4. Insert action is successful and data with values(666, 'data666') is missing after compact success.
### What doesn't meet your expectations?
The insert data with values(666, 'data666') should not be lost, It's better to throw a compact error instead of making data lose
### Anything else?
I checked the code, the compact action just implemented same as the overwrite action, and system always uses the latest snapshot to mark all the files as deleted instead of using the snapshot which the compact action invoked. Currently, the compact/overwrite action only obeys SNAPSHOT isolation instead of SERIALIZABLE isolation. I am not sure if it is as expected? however, current implement of compaction is very dangerous in our scene.
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Spark SQL reproduction and the compact procedure, then inspect the compact and overwrite action paths that mark files deleted and handle snapshots. Trace how a concurrent insert is treated during compaction. Done means the new record is retained or compaction fails safely, with a regression test covering the concurrent operation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100