apache / apache/paimon

[Bug]

Open
#3,520 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8

### Compute Engine

flink1.17.1

### Minimal reproduce step

CREATE TABLE test (
`id` BIGINT ,
`user_id` BIGINT ,
`user_source` BIGINT ,
`identity` BIGINT ,
`friend_id` BIGINT ,
`friend_source` BIGINT ,
`platform` BIGINT ,
`delete_type` BIGINT,
`add_time` STRING ,
`ums_ts_` BIGINT )
USING paimon
COMMENT 'test'
TBLPROPERTIES(
'EXTERNAL' = 'TRUE',
'bucket' = '600',
'bucketing_version' = '2',
'changelog-producer' = 'input',
'consumer.expiration-time' = '30m',
'file.format' = 'parquet',
'metastore.partitioned-table' = 'true',
'path' = 'hdfs:///tmp/paimon/test',
'primary-key' = 'id,user_id,friend_id',
'sequence.field' = 'ums_ts_',
'storage_handler' = 'org.apache.paimon.hive.PaimonStorageHandler',
'write-only' = 'true',
'write.merge-schema.explicit-cast' = 'true');

Our PK table is sharded, so we specified the primary key with three fields: 'primary-key' = 'id,user_id,friend_id'. After running for a while, the compaction task reports an error: Caused by: java.io.IOException: java.lang.IllegalStateException: SortedRun is not sorted and may contain overlapping key intervals. This is a bug.

We found that some data has duplicate id values, even though we specified three fields as the primary key. Additionally, we discovered that _KEY_id is NULL, although _KEY_id should theoretically have the same value as the id field. In what situations would _KEY_id be NULL? I believe the above error is caused by _KEY_id being NULL, but I don't understand the reasons or scenarios that lead to this. How should I troubleshoot this issue?

![image](https://github.com/apache/paimon/assets/10754296/be36704c-f596-4b31-9add-e04dd590e8c6)

![lQLPJxPYSQxREgnMhM0FirCEzwychiSL_wZW9zYAuEAA_1418_132](https://github.com/apache/paimon/assets/10754296/d83b0970-40d8-4654-9d35-652f5f18e1d7)

### What doesn't meet your expectations?

How should I troubleshoot this issue?

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Start by reproducing the provided CREATE TABLE case on Paimon 0.8 with Flink 1.17.1, then inspect the compaction failure and compare the declared three-field primary key with rows where _KEY_id is NULL. Done means identifying the cause or scenario and documenting a concrete troubleshooting path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.