Optimize scanInternalv2 to single pass
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The current algorithm take two passes over the log blocks: # First pass to collect all the valid blocks alongwith block instant times including rollback block's target instant time.
# Second pass, in rever order of block instant time, to track final compacted instant times for each block.
Now that we have removed appending to the same log file for multiple deltacommits, we can probably scan in single pass by keeping an active list or hash map of block times to their corresponding block, updating as we go. Should be tested for:
# Out of order merged blocks: Log compaction is scheduled and by the time it appended a block, another block is added by another writer.
# Log compaction operation failed, so a rollback is issued for this block. Here the rollback can be next block or can come at a later point of time.
# Log compaction is executing and, before committing, compaction starts running on the same file group.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6888
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-3580
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.