[Bug] Spark Batch dynamic overwrite encountered a file deletion conflict exception
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5
### Compute Engine
Spark
### Minimal reproduce step
Environment
Spark3.2.1
Metastore Client2.3.9
Hive3.1.0
Step1
create table paimon.test.mytable_v13 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
stat_date STRING
) USING paimon PARTITIONED BY (stat_date) tblproperties (
'primary-key' = 'stat_date,user_id',
'metastore.partitioned-table'='true',
'changelog-producer'='lookup',
'streaming-read-overwrite'='true'
);
insert into paimon.test.mytable_v12 values(1,1,'pv','20231016');
select * from paimon.test.mytable_v12;

Step2
create table paimon.test.mytable_v14 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
stat_date STRING
) USING paimon tblproperties (
'primary-key' = 'stat_date,user_id',
'metastore.partitioned-table'='true',
'changelog-producer'='lookup',
'streaming-read-overwrite'='true'
);
insert into paimon.test.mytable_v14 values(1,2,'pv','20231016');
select * from paimon.test.mytable_v14;

Step3
INSERT OVERWRITE paimon.test.mytable_v13 SELECT * FROM paimon.test.mytable_v14;

### What doesn't meet your expectations?
Overwrite should not conflict
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the three-step table setup and INSERT OVERWRITE using Spark 3.2.1, Hive 3.1.0, and metastore client 2.3.9. Trace the Spark batch dynamic-overwrite path to determine why file deletion conflicts occur, then verify that the overwrite completes without conflict and produces the expected data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100