Avoid reading log files for resolving schema for _hoodie_operation field
Open
from-jira
priority:high
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The table schema resolver needs to read schema from the data files (base or log files) to see whether _hoodie_operation field is present for Flink CDC use cases. This can cause overhead of reading data file footers multiple times. We should see if we can store a table config to indicate if _hoodie_operation field is present in the table, or simplify the Flink CDC format in Hudi 1.0 (thus no need of _hoodie_operation field and schema resolver).
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7585
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-6242
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.