HoodieParquetDataBlock failed to read data when schema evolution was enabled
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
When scheme evolution is enabled and a column is renamed, the parquet datablock fails to be read because the schema does not match
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5199
- Type: Bug
- Affects version(s):
- 0.12.1
---
## Comments
11/Nov/22 19:21;tangshangwen;When the table schema changes, we should use parquet's WriterSchema to read the file;;;
---
11/Nov/22 19:32;tangshangwen;I will submit PR to fix this problem;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieParquetDataBlock and reproduce the failure with schema evolution and a renamed column. Read how Parquet's WriterSchema is used for reading, then verify that the datablock reads successfully when the table schema changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100