apache / apache/paimon

[Bug] Query result duplicate primary key

Open
#3,841 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.7.0-incubating

### Compute Engine

Flink 1.18.0

### Minimal reproduce step

we have a flink job write data to paimon table,table options is:
+---------------------------+---------+
| key | value |
+---------------------------+---------+
| bucket | 8 |
| scan.remove-normalize | true |
| deduplicate.ignore-delete | true |
| changelog-producer | none |
| file.format | parquet |
+---------------------------+---------+

### What doesn't meet your expectations?

When job execute for some time, we use batch mode query table ,some of our query results duplicate primary key.
![image](https://github.com/user-attachments/assets/181667c7-3ec5-4126-b15a-d86ec17a8774)
When we update paimon version to query this table,it also has duplicate primary key.

### Anything else?

I want to know what cause this problem. Is this problem caused by writer operator? Does a later version fix this issue?

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the duplicate-primary-key result with Paimon 0.7.0-incubating, Flink 1.18.0, batch mode, and the table options listed in the issue. Compare the result with a later Paimon version and trace whether the cause is in the writer or query path; done means identifying the responsible component and affected versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.