apache / apache/paimon

[Bug] using Chain table group partition ,partition in snapshot but not in delta will not be read

Open
#7,532 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

i tried using chain group parition to solve my #7503 , but still failed read some data

### Compute Engine

spark 3.5.2

### Minimal reproduce step

using test case :
```
CREATE TABLE IF NOT EXISTS `default`.`chain_test` (
`t1` BIGINT COMMENT 't1',
`t2` BIGINT COMMENT 't2',
`t3` STRING COMMENT 't3'
) using paimon PARTITIONED BY (`region` STRING, `dt` STRING COMMENT 'dt')
TBLPROPERTIES (
'bucket-key' = 't1',
'primary-key' = 'region,dt,t1',
'partition.timestamp-pattern' = '$dt',
'partition.timestamp-formatter' = 'yyyyMMdd',
'chain-table.enabled' = 'true',
'bucket' = '2',
'merge-engine' = 'deduplicate',
'sequence.field' = 't2',
'chain-table.chain-partition-keys' = 'dt'
);

CALL sys.create_branch('default.chain_test', 'snapshot');
CALL sys.create_branch('default.chain_test', 'delta');

ALTER TABLE default.chain_test SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);

ALTER TABLE `default`.`chain_test$branch_snapshot` SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);

ALTER TABLE `default`.`chain_test$branch_delta` SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);
```
insert some data
```
-- 写入 delta 分支
SET spark.paimon.branch = delta;

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'CN', dt = '20250809')
VALUES (1, 1, '1'), (2, 1, '1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'CN', dt = '20250810')
VALUES (1, 2, '1-1'), (3, 1, '1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'CN', dt = '20250811')
VALUES (2, 2, '1-1'), (4, 1, '1');
VALUES (5, 2, '1-1'), (6, 2, '1-1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'US', dt = '20250809')
VALUES (11, 1, '1'), (12, 1, '1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'US', dt = '20250810')
VALUES (11, 2, '1-1'), (13, 1, '1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'US', dt = '20250811')
VALUES (12, 2, '1-1'), (14, 1, '1');

-- 写入 snapshot 分支
SET spark.paimon.branch = snapshot;

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'CN', dt = '20250810')
VALUES (1, 2, '1-1'), (2, 1, '1'), (3, 1, '1');

INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'US', dt = '20250810')
VALUES (11, 2, '1-1'), (12, 1, '1'), (13, 1, '1');

```
query
```
SELECT * FROM `default`.`chain_test` where dt = '20250811'
```

result
```
1 2 1-1 CN 20250811
2 2 1-1 CN 20250811
4 1 1 CN 20250811
3 1 1 CN 20250811
12 2 1-1 US 20250811
11 2 1-1 US 20250811
13 1 1 US 20250811
14 1 1 US 20250811
```
this is ok, why? because all partition are in all branch

if i insert a partition not in delta
```
SET spark.paimon.branch = snapshot;
INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'UK', dt = '20250810')
VALUES (1, 2, '1-1'), (2, 1, '1'), (3, 1, '1');
```
still got query result
```
1 2 1-1 CN 20250811
2 2 1-1 CN 20250811
4 1 1 CN 20250811
3 1 1 CN 20250811
12 2 1-1 US 20250811
11 2 1-1 US 20250811
13 1 1 US 20250811
14 1 1 US 20250811
```

### What doesn't meet your expectations?

when insert a partition not in delta ,and in snapshot
```
INSERT OVERWRITE TABLE `default`.`chain_test` PARTITION (region = 'UK', dt = '20250810')
VALUES (1, 2, '1-1'), (2, 1, '1'), (3, 1, '1');
```

query result should be
```
1 2 1-1 CN 20250811
2 2 1-1 CN 20250811
4 1 1 CN 20250811
3 1 1 CN 20250811
12 2 1-1 US 20250811
11 2 1-1 US 20250811
13 1 1 US 20250811
14 1 1 US 20250811
1 2 1-1 UK 20250811
2 1 1 UK 20250811
3 1 1 UK 20250811
```

or i try to change partition key order from `PARTITIONED BY (`region` STRING, `dt` STRING COMMENT 'dt')` to `PARTITIONED BY (`dt` STRING COMMENT 'dt',`region` STRING)`

when execute sql as above
result is
```
2 2 1-1 20250811 US
4 1 1 20250811 US
12 2 1-1 20250811 US
14 1 1 20250811 US
2 2 1-1 20250811 CN
4 1 1 20250811 CN

```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the provided Spark SQL reproduction with chain-table partitioning and separate snapshot and delta branches, then compare reads for partitions present in only one branch. Trace the partition-selection behavior for `dt = '20250811'`; done means the query also returns the expected UK rows from the snapshot branch without changing results for CN and US.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.