Fetch schema from log file while computing col stats
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We are using table schema resolver to fetch writer schema to read log file while computing col stats. Ref : [https://github.com/apache/hudi/pull/12105]
Lets follow up to see if we can just fetch the schema from the log file directly rather than using table schema resolver.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8445
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-8727
- Fix version(s):
- 1.1.0
---
## Comments
28/Oct/24 17:16;yihua;The log blocks and files can be written by different commits using different writer schemas in case there is schema evolution. Strictly speaking, the column stats of a log file should only contain the columns that exist in the file. Using the table schema in this case may introduce additional column stats entries of columns that do not exist in the log file, making it hard to understand.;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.