HoodieNativeAvroHFileReader throws ZipException on valid column_stats HFiles during MDT compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug description
After upgrading from Hudi 0.14.x to 1.0.2, the default `HoodieNativeAvroHFileReader` (enabled via `hoodie.metadata.reader.use.native.reader=true`, which is the 1.0.2 default) throws a `java.util.zip.ZipException` when reading certain `column_stats` partition HFiles in the metadata table (MDT) during compaction. The same HFiles read successfully when using the legacy `HoodieAvroHFileReader` (`hoodie.metadata.reader.use.native.reader=false`).
The failure occurs deterministically on specific HFiles — not all column_stats HFiles trigger it. The affected HFiles are byte-valid (not corrupted on storage) and were written by Hudi 1.0.2 writers with no errors at write time. The crash surfaces during the next MDT compaction that reads the affected HFile block.
### Impact
The `ZipException` crash-loops the writer process on every subsequent write attempt, because each write triggers MDT compaction which re-reads the same corrupted block. This makes the affected table unwritable until the workaround is applied.
### Steps to reproduce
1. Use Hudi 1.0.2 with `column_stats` metadata partition enabled (default).
2. Write to a COW table with multiple concurrent writers using OCC (DynamoDB-based lock provider) over an extended period.
3. At some point, an MDT compaction reads a `column_stats` HFile block that triggers the `ZipException`.
**Note:** The issue is intermittent — it depends on the specific data/block layout written to the HFile. Not every `column_stats` HFile triggers it.
### Expected behavior
`HoodieNativeAvroHFileReader` should read all HFiles that `HoodieAvroHFileReader` can read without error. Both readers operate on the same byte-identical files.
### Actual behavior
```
java.util.zip.ZipException:
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
at org.apache.hadoop.hbase.io.compress.HFileDecompressor.decompress(...)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(...)
at org.apache.hudi.io.storage.HoodieNativeAvroHFileReader.readBlock(...)
...
```
The legacy `HoodieAvroHFileReader` reads the same file cleanly.
### Diagnosis
A/B testing confirmed the issue is isolated to the reader implementation:
- **`hoodie.metadata.reader.use.native.reader=true`** (native, 1.0.2 default): `ZipException` on the affected HFile block
- **`hoodie.metadata.reader.use.native.reader=false`** (legacy): reads all records successfully (confirmed >9M records read without error from the same file)
The native reader's block decompression logic in `HoodieNativeAvroHFileReader` mis-handles certain HFile block encodings that the HBase-backed legacy reader handles correctly. The root cause appears to be in the native reader's block unpacking/decompression path, not in the data written to the HFile.
### Workaround
Set `hoodie.metadata.reader.use.native.reader=false` in the writer configuration to fall back to the legacy `HoodieAvroHFileReader`. This avoids the crash with no functional difference (same data, same MDT behavior).
### Environment
- **Hudi version:** 1.0.2
- **Spark version:** 3.5.x
- **Java version:** JDK 17
- **Storage:** S3
- **Table type:** COW
- **MDT partitions enabled:** `files`, `column_stats`
- **Lock provider:** DynamoDB-based OCC
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at HoodieNativeAvroHFileReader.readBlock and compare its block unpacking and decompression path with the legacy HoodieAvroHFileReader on the same column_stats HFiles. Reproduce with hoodie.metadata.reader.use.native.reader=true, then verify that affected HFiles read without ZipException while the legacy reader remains equivalent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100