SPJ with Bucket Partition Key: Error: Wrong class, expected java.lang.CharSequence, but was java.lang.Integer
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
Tried both:
- iceberg-spark-runtime-3.5_2.12-1.10.1.jar
- /usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar (from AWS EMR)
### Query engine
Spark 3.5.6
### Please describe the bug 🐞
I have a target table & input table, both are written in Iceberg. Based on the manifest, the partition spec of target table is:
```
"partition-specs":[{"spec-id":0,"fields":[{"name":"collected_date","transform":"identity","source-id":29,"field-id":1000},{"name":"user_id_bucket","transform":"bucket[32]","source-id":2,"field-id":1001}]}]
```
And partition spec of input table is
```
"partition-specs":[{"spec-id":0,"fields":[{"name":"collected_date","transform":"identity","source-id":29,"field-id":1000},{"name":"user_id_bucket","transform":"bucket[32]","source-id":2,"field-id":1001}]}]
```
The field `user_id` is originally a string column, but the bucketed value turns to Integer. I'm running SPJ by doing a `MERGE INTO` with join on `user_id`
```
MERGE INTO target AS t
USING source AS s
ON t.post_id = s.post_id AND t.user_id = s.user_id
WHEN MATCHED AND (t.collected_at IS NULL OR t.collected_at <= s.collected_at)
THEN UPDATE SET *
WHEN NOT MATCHED
THEN INSERT *
```
The `MERGE INTO` fails with this error
```
pyspark.errors.exceptions.captured.IllegalArgumentException: Wrong class, expected java.lang.CharSequence, but was java.lang.Integer, for object: 1
```
Here's the trace
```
2026-02-17 15:06:12,666 - job - ERROR - Error: Wrong class, expected java.lang.CharSequence, but was java.lang.Integer, for object: 1
Traceback (most recent call last):
File "/mnt/yarn/usercache/hadoop/appcache/application_1771340623606_0001/container_1771340623606_0001_01_000001/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
File "/mnt/yarn/usercache/hadoop/appcache/application_1771340623606_0001/container_1771340623606_0001_01_000001/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/mnt/yarn/usercache/hadoop/appcache/application_1771340623606_0001/container_1771340623606_0001_01_000001/pyspark.zip/pyspark/errors/exceptions/captured.py", line 185, in deco
pyspark.errors.exceptions.captured.IllegalArgumentException: Wrong class, expected java.lang.CharSequence, but was java.lang.Integer, for object: 1
```
Both target & source are stored with `format-version`=`2`
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by reproducing the MERGE INTO query with Spark 3.5.6 using matching bucket partition specs and the Iceberg runtime versions listed. Trace SPJ handling of the string user_id and its integer bucket value; done means the merge completes without the CharSequence-versus-Integer error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100