Getting `Caused by: java.lang.UnsupportedOperationException` when running a previously successful Spark Job on writing to Hudi
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Getting the following error when trying to run a spark job which reads and upserts a large amount of data into a hudi table.
```
org.apache.hudi.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
reserved (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
maxValue (org.apache.hudi.avro.model.HoodieMetadataColumnStats)
columnStatMetadata (org.apache.hudi.metadata.HoodieMetadataPayload)
data (org.apache.hudi.common.model.HoodieAvroRecord)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[__app__.jar:?]
...
Caused by: java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) ~[?:1.8.0_382]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[__app__.jar:?]
... 55 more
```
Getting it at the following job.stage
```
Preparing compaction metadata: redactedtablename_metadata
collect at HoodieJavaRDD.java:155
```
**To Reproduce**
Reproduction may be difficult. The job successfully ran at least 7 times before beginning to have this error.
Steps to reproduce the behavior:
2. Create a spark job which write to the Hudi.table
3. Execute the spark job
**Expected behavior**
The hudi write should successfully execute
**Environment Description**
* Hudi version :
```
0.12.2
```
* Spark version :
```
3.3
3.3.0
```
* Hive version :
```
2.12
spark-hive_${scala.binary.version}
```
* Hadoop version :
```
3.3.3
```
* Storage (HDFS/S3/GCS..) :
S3
* Running on Docker? (yes/no) :
No, AWS.EMR
**Stacktrace**
```
org.apache.hudi.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
reserved (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
maxValue (org.apache.hudi.avro.model.HoodieMetadataColumnStats)
columnStatMetadata (org.apache.hudi.metadata.HoodieMetadataPayload)
data (org.apache.hudi.common.model.HoodieAvroRecord)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[__app__.jar:?]
...
Caused by: java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) ~[?:1.8.0_382]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[__app__.jar:?]
... 55 more
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the stack trace and the collect call at HoodieJavaRDD.java:155, then inspect HoodieMetadataPayload, HoodieAvroRecord, and HoodieMetadataColumnStats around the reported serialization path. Try to reproduce the failure with Hudi 0.12.2, Spark 3.3.0, and AWS EMR, and identify the conditions that make metadata preparation fail. Done means the write completes successfully without the Kryo UnsupportedOperationException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100