apache / apache/hudi

Fetch schema from log file while computing col stats

Open
#17,301 1 comment 0 reactions 1 assignee Assigned to @nsivabalan View on GitHub
area:metadata-table from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We are using table schema resolver to fetch writer schema to read log file while computing col stats. Ref : [https://github.com/apache/hudi/pull/12105] 

 

Lets follow up to see if we can just fetch the schema from the log file directly rather than using table schema resolver. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8445
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-8727
- Fix version(s):
- 1.1.0

---

## Comments

28/Oct/24 17:16;yihua;The log blocks and files can be written by different commits using different writer schemas in case there is schema evolution.  Strictly speaking, the column stats of a log file should only contain the columns that exist in the file.  Using the table schema in this case may introduce additional column stats entries of columns that do not exist in the log file, making it hard to understand.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.