apache / apache/paimon

[Bug] Spark Batch dynamic overwrite encountered a file deletion conflict exception

Open
#2,164 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.5

### Compute Engine

Spark

### Minimal reproduce step

Environment
Spark3.2.1
Metastore Client2.3.9
Hive3.1.0

Step1
create table paimon.test.mytable_v13 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
stat_date STRING
) USING paimon PARTITIONED BY (stat_date) tblproperties (
'primary-key' = 'stat_date,user_id',
'metastore.partitioned-table'='true',
'changelog-producer'='lookup',
'streaming-read-overwrite'='true'
);
insert into paimon.test.mytable_v12 values(1,1,'pv','20231016');
select * from paimon.test.mytable_v12;
![image](https://github.com/apache/incubator-paimon/assets/75019445/15e1a959-b28b-4823-8c6f-a54b379c7a88)

Step2
create table paimon.test.mytable_v14 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
stat_date STRING
) USING paimon tblproperties (
'primary-key' = 'stat_date,user_id',
'metastore.partitioned-table'='true',
'changelog-producer'='lookup',
'streaming-read-overwrite'='true'
);
insert into paimon.test.mytable_v14 values(1,2,'pv','20231016');
select * from paimon.test.mytable_v14;
![image](https://github.com/apache/incubator-paimon/assets/75019445/573136ff-f5ba-466c-ba9b-2cca5ade87b1)

Step3
INSERT OVERWRITE paimon.test.mytable_v13 SELECT * FROM paimon.test.mytable_v14;
![image](https://github.com/apache/incubator-paimon/assets/75019445/30498686-0b95-4f77-beeb-49887aa0d135)

### What doesn't meet your expectations?

Overwrite should not conflict

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the three-step table setup and INSERT OVERWRITE using Spark 3.2.1, Hive 3.1.0, and metastore client 2.3.9. Trace the Spark batch dynamic-overwrite path to determine why file deletion conflicts occur, then verify that the overwrite completes without conflict and produces the expected data.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.