[SUPPORT] Table with delete block written by 0.13.x will not be readable by lower versions, such as 0.12.x
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
https://github.com/apache/hudi/pull/7024 change the serialization of uft8, which cause table with delete block will not be readable by lower versions, such as 0.12.x
```text
Caused by: org.apache.hudi.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 27
Serialization trace: orderingVal (org.apache.hudi.common.model.DeleteRecord)
at org.apache.hudi.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391)
at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:100)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:74)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.deserialize(HoodieDeleteBlock.java:106)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.getRecordsToDelete(HoodieDeleteBlock.java:91)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:473)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:343) ...
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked PR #7024 and the stack-trace entry points in HoodieDeleteBlock, AbstractHoodieLogRecordReader, SerializationUtils, and DeleteRecord. Reproduce reading a delete block written by 0.13.x with 0.12.x, then determine the compatibility behavior required and verify that the lower version can read the table without the Kryo class-ID failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100