apache / apache/hudi

Add an iterator api for HoodieUnMergedLogRecordScanner

Open
#16,599 2 comments 0 reactions 1 assignee Assigned to @danny0405 View on GitHub
area:reader from-jira priority:critical status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

HoodieUnMergedLogRecordScanner extends from 
AbstractHoodieLogRecordReader, the later is used for merging multiple log files within one file slice. While for HoodieUnMergedLogRecordScanner, there is no merging, all we need to do is resolving the hoodie record and return it back.

The current impl is kind of hacky, it allows the HoodieUnMergedLogRecordScanner to pass around a 
LogRecordScannerCallback, each resolved record in applied with this callback, the singular callback impl in Flink side is just to put the record into another queue, and pop the records again with this queue.

Let's refactor this:

1. add a iterator API for HoodieUnMergedLogRecordScanner, there might need some code refactoring also to the AbstractHoodieLogRecordReader to make the code more de-coupled;
2. in Flink side, use this new iterator instead to decipher the records.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8163
- Type: Improvement
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.