apache / apache/paimon

[Bug] ClassCastException: BinaryString to Integer during shuffle on integer-type partition column in Spark Reader

Open
#6,440 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.2.0

### Compute Engine

spark 3.5

### Minimal reproduce step

1. create table
```
CREATE TABLE main.test (
order_id STRING,
order_price DOUBLE,
order_date STRING,
order_hour INT
)
USING paimon
PARTITIONED BY (order_date, order_hour)
TBLPROPERTIES (
'bucket-key' = 'order_id',
'bucket' = '16',
'primary-key' = 'order_date,order_hour,order_id',
'deletion-vectors.enabled' = 'true'
);
```

2. insert some record
```
INSERT INTO main.test VALUES ('order_001', 1.0, '2025-10-10', 11)
```

3. run a query with `DISTINCT`
```
SELECT DISTINCT order_hour FROM main.test
```

### What doesn't meet your expectations?

Resulting
```
Py4JJavaError: An error occurred while calling o154.showString.
: java.lang.ClassCastException: class org.apache.paimon.data.BinaryString cannot be cast to class java.lang.Integer (org.apache.paimon.data.BinaryString is in unnamed module of loader org.apache.spark.util.MutableURLClassLoader @989da1; java.lang.Integer is in module java.base of loader 'bootstrap')
```

### Anything else?

If I modify partition columns to
```
order_date DATE,
order_hour INT
```
the above error will not happen, which seems a little weird to me.

But when using `DATE` type in partition columns, I will encounter another issue: [[Bug] expire_partitions procedure fails on DATE type partition columns](https://github.com/apache/paimon/issues/6439)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with Spark 3.5 using the provided Paimon table definition and SELECT DISTINCT order_hour query. Compare the behavior of STRING and DATE partition columns, and use the related expire_partitions issue for context; done means the integer partition value no longer causes a BinaryString-to-Integer ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.