apache / apache/hudi

Getting `Caused by: java.lang.UnsupportedOperationException` when running a previously successful Spark Job on writing to Hudi

Open
#9,443 4 comments 0 reactions 0 assignees View on GitHub
area:metadata-table area:writer component:kryo priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Getting the following error when trying to run a spark job which reads and upserts a large amount of data into a hudi table.

```
org.apache.hudi.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
reserved (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
maxValue (org.apache.hudi.avro.model.HoodieMetadataColumnStats)
columnStatMetadata (org.apache.hudi.metadata.HoodieMetadataPayload)
data (org.apache.hudi.common.model.HoodieAvroRecord)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[__app__.jar:?]

...

Caused by: java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) ~[?:1.8.0_382]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[__app__.jar:?]
... 55 more
```

Getting it at the following job.stage
```
Preparing compaction metadata: redactedtablename_metadata
collect at HoodieJavaRDD.java:155
```

**To Reproduce**

Reproduction may be difficult. The job successfully ran at least 7 times before beginning to have this error.

Steps to reproduce the behavior:

2. Create a spark job which write to the Hudi.table
3. Execute the spark job

**Expected behavior**

The hudi write should successfully execute

**Environment Description**

* Hudi version :
```
0.12.2
```

* Spark version :
```
3.3
3.3.0
```

* Hive version :

```
2.12

spark-hive_${scala.binary.version}
```

* Hadoop version :
```
3.3.3
```

* Storage (HDFS/S3/GCS..) :

S3

* Running on Docker? (yes/no) :

No, AWS.EMR

**Stacktrace**

```

org.apache.hudi.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
reserved (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
maxValue (org.apache.hudi.avro.model.HoodieMetadataColumnStats)
columnStatMetadata (org.apache.hudi.metadata.HoodieMetadataPayload)
data (org.apache.hudi.common.model.HoodieAvroRecord)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[__app__.jar:?]

...

Caused by: java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) ~[?:1.8.0_382]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[__app__.jar:?]
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[__app__.jar:?]
... 55 more
```

Screenshot 2023-08-14 at 10 40 25 AM

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the stack trace and the collect call at HoodieJavaRDD.java:155, then inspect HoodieMetadataPayload, HoodieAvroRecord, and HoodieMetadataColumnStats around the reported serialization path. Try to reproduce the failure with Hudi 0.12.2, Spark 3.3.0, and AWS EMR, and identify the conditions that make metadata preparation fail. Done means the write completes successfully without the Kryo UnsupportedOperationException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.