apache / apache/hudi

[SUPPORT] Table with delete block written by 0.13.x will not be readable by lower versions, such as 0.12.x

Open
#9,195 2 comments 0 reactions 0 assignees View on GitHub
issue:version-compatibility priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

https://github.com/apache/hudi/pull/7024 change the serialization of uft8, which cause table with delete block will not be readable by lower versions, such as 0.12.x

```text
Caused by: org.apache.hudi.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 27
Serialization trace: orderingVal (org.apache.hudi.common.model.DeleteRecord)
at org.apache.hudi.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:100)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:74)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.deserialize(HoodieDeleteBlock.java:106)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.getRecordsToDelete(HoodieDeleteBlock.java:91)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:473)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:343) ...
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked PR #7024 and the stack-trace entry points in HoodieDeleteBlock, AbstractHoodieLogRecordReader, SerializationUtils, and DeleteRecord. Reproduce reading a delete block written by 0.13.x with 0.12.x, then determine the compatibility behavior required and verify that the lower version can read the table without the Kryo class-ID failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.