[Bug]
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8
### Compute Engine
flink1.17.1
### Minimal reproduce step
CREATE TABLE test (
`id` BIGINT ,
`user_id` BIGINT ,
`user_source` BIGINT ,
`identity` BIGINT ,
`friend_id` BIGINT ,
`friend_source` BIGINT ,
`platform` BIGINT ,
`delete_type` BIGINT,
`add_time` STRING ,
`ums_ts_` BIGINT )
USING paimon
COMMENT 'test'
TBLPROPERTIES(
'EXTERNAL' = 'TRUE',
'bucket' = '600',
'bucketing_version' = '2',
'changelog-producer' = 'input',
'consumer.expiration-time' = '30m',
'file.format' = 'parquet',
'metastore.partitioned-table' = 'true',
'path' = 'hdfs:///tmp/paimon/test',
'primary-key' = 'id,user_id,friend_id',
'sequence.field' = 'ums_ts_',
'storage_handler' = 'org.apache.paimon.hive.PaimonStorageHandler',
'write-only' = 'true',
'write.merge-schema.explicit-cast' = 'true');
Our PK table is sharded, so we specified the primary key with three fields: 'primary-key' = 'id,user_id,friend_id'. After running for a while, the compaction task reports an error: Caused by: java.io.IOException: java.lang.IllegalStateException: SortedRun is not sorted and may contain overlapping key intervals. This is a bug.
We found that some data has duplicate id values, even though we specified three fields as the primary key. Additionally, we discovered that _KEY_id is NULL, although _KEY_id should theoretically have the same value as the id field. In what situations would _KEY_id be NULL? I believe the above error is caused by _KEY_id being NULL, but I don't understand the reasons or scenarios that lead to this. How should I troubleshoot this issue?


### What doesn't meet your expectations?
How should I troubleshoot this issue?
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files or tests are named. Start by reproducing the provided CREATE TABLE case on Paimon 0.8 with Flink 1.17.1, then inspect the compaction failure and compare the declared three-field primary key with rows where _KEY_id is NULL. Done means identifying the cause or scenario and documenting a concrete troubleshooting path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100