apache / apache/fluss

[lake] Union read on partitioned table returns duplicated rows when partition column is non-String type

Open
#3,490 1 comment 0 reactions 1 assignee Claimed by @bryndenZh View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main

### Please describe the bug 🐞

For a partitioned table with `table.datalake.enabled = true` and a **non-`STRING` partition column** (`DATE`, `INT`, `TIMESTAMP`, etc.), after data is tiered to the lake the default union read returns each row twice (once from the lake, once from the Fluss log), so the row count doubles. Affects both log tables and primary-key tables.

#### Reproduce

```sql
CREATE TABLE log_part_date (id BIGINT, dt DATE) PARTITIONED BY (dt)
WITH ('table.datalake.enabled' = 'true', 'table.datalake.freshness' = '10s');
INSERT INTO log_part_date VALUES (1, DATE '2024-03-01');
-- after tiering commits a snapshot, in batch mode:
SELECT COUNT(*) FROM log_part_date; -- expected 1, actual 2
SELECT COUNT(*) FROM log_part_date$lake; -- 1
```

#### Root cause

[`PaimonSplit.partition()`](https://github.com/apache/fluss/blob/main/fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/source/PaimonSplit.java#L54-L67) reads every partition field via `BinaryRow.getString(i)` regardless of logical type, producing garbage for non-`STRING` columns. The lake-side partition name then never matches the Fluss-side name in `LakeSplitGenerator.generatePartitionTableSplit`, so the same partition is emitted as both a lake split and a Fluss-log split.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.