apache / apache/hudi

Refactor TableSchemaResolver to remove `includeMetadataFields` flags

Open
#15,063 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

Currently, `TableSchemaResolver` to pass `includeMetadataFields` in its APIs that would selectively remove metadata fields from the returned schemas. There are multiple issues with this flag:
# It's applied inconsistently: sometimes it just means that meta fields {_}won't be added{_}, and sometimes it means fields _would be removed_ even if present
# This flag actually spells the usage context into TableSchemaResolver: it's highly contextual whether caller wants to remove, omit such meta-fields, or take the schema as is and should not be spilled into the Resolver itself.
# B/c of it there's no way to know actually what was actual schema the data was written with (b/c flag might not only omit, but also change the original schema)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3626
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.