[Bug] java.lang.ClassCastException: org.apache.paimon.spark.SparkInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.7
### Compute Engine
spark 3.1.2 - 2.12
### Minimal reproduce step
`24/08/16 10:58:18 WARN TaskSetManager: Lost task 4.0 in stage 6.0 (TID 584) (emr-worker-41874.cluster-215453 executor 10): java.lang.ClassCastException: org.apache.paimon.spark.SparkInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
at org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$1.writeValue(UnsafeRowSerializer.scala:64)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:249)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:158)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
24/08/16 10:58:20 WARN TaskSetManager: Lost task 21.0 in stage 6.0 (TID 601) (emr-worker-41874.cluster-215453 executor 10): java.lang.ClassCastException
`
### What doesn't meet your expectations?
-
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with Paimon 0.7 on Spark 3.1.2 with Scala 2.12, using the reported shuffle stack trace as the entry point. Inspect the interaction between SparkInternalRow and UnsafeRowSerializer; done means the shuffle completes without the ClassCastException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100