[lake] Union read on partitioned table returns duplicated rows when partition column is non-String type
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main
### Please describe the bug 🐞
For a partitioned table with `table.datalake.enabled = true` and a **non-`STRING` partition column** (`DATE`, `INT`, `TIMESTAMP`, etc.), after data is tiered to the lake the default union read returns each row twice (once from the lake, once from the Fluss log), so the row count doubles. Affects both log tables and primary-key tables.
#### Reproduce
```sql
CREATE TABLE log_part_date (id BIGINT, dt DATE) PARTITIONED BY (dt)
WITH ('table.datalake.enabled' = 'true', 'table.datalake.freshness' = '10s');
INSERT INTO log_part_date VALUES (1, DATE '2024-03-01');
-- after tiering commits a snapshot, in batch mode:
SELECT COUNT(*) FROM log_part_date; -- expected 1, actual 2
SELECT COUNT(*) FROM log_part_date$lake; -- 1
```
#### Root cause
[`PaimonSplit.partition()`](https://github.com/apache/fluss/blob/main/fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/source/PaimonSplit.java#L54-L67) reads every partition field via `BinaryRow.getString(i)` regardless of logical type, producing garbage for non-`STRING` columns. The lake-side partition name then never matches the Fluss-side name in `LakeSplitGenerator.generatePartitionTableSplit`, so the same partition is emitted as both a lake split and a Fluss-log split.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.