apache / apache/hudi

Validate if hfile backward compatibility

Open
#17,431 0 comments 0 reactions 1 assignee Assigned to @linliu-code View on GitHub
from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

It means that:

The hfile written through our native writer from 1.1 release can be read by hbase reader from 0.x and 1.0.2 release.

Therefore, we need to generate the hfile, and validates if they can be read successfully by different readers.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9449
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-8222
- Fix version(s):
- 1.1.0

---

## Comments

24/May/25 22:55;yihua;An alternative approach is to make sure the bytes written in the HFile are identical between HBase and native HFile writer.  We should a test for such a storage byte validation.;;;

---

09/Sep/25 20:14;linliu;Generally we could attempt this byte by byte approach; we may face some issues like encoding / compression difference, index level setting difference, MVCC byte, etc; but generally we can certify majority of a file should be identical.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.