[Bug] ClassCastException: BinaryString to Integer during shuffle on integer-type partition column in Spark Reader
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.2.0
### Compute Engine
spark 3.5
### Minimal reproduce step
1. create table
```
CREATE TABLE main.test (
order_id STRING,
order_price DOUBLE,
order_date STRING,
order_hour INT
)
USING paimon
PARTITIONED BY (order_date, order_hour)
TBLPROPERTIES (
'bucket-key' = 'order_id',
'bucket' = '16',
'primary-key' = 'order_date,order_hour,order_id',
'deletion-vectors.enabled' = 'true'
);
```
2. insert some record
```
INSERT INTO main.test VALUES ('order_001', 1.0, '2025-10-10', 11)
```
3. run a query with `DISTINCT`
```
SELECT DISTINCT order_hour FROM main.test
```
### What doesn't meet your expectations?
Resulting
```
Py4JJavaError: An error occurred while calling o154.showString.
: java.lang.ClassCastException: class org.apache.paimon.data.BinaryString cannot be cast to class java.lang.Integer (org.apache.paimon.data.BinaryString is in unnamed module of loader org.apache.spark.util.MutableURLClassLoader @989da1; java.lang.Integer is in module java.base of loader 'bootstrap')
```
### Anything else?
If I modify partition columns to
```
order_date DATE,
order_hour INT
```
the above error will not happen, which seems a little weird to me.
But when using `DATE` type in partition columns, I will encounter another issue: [[Bug] expire_partitions procedure fails on DATE type partition columns](https://github.com/apache/paimon/issues/6439)
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with Spark 3.5 using the provided Paimon table definition and SELECT DISTINCT order_hour query. Compare the behavior of STRING and DATE partition columns, and use the related expire_partitions issue for context; done means the integer partition value no longer causes a BinaryString-to-Integer ClassCastException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100