apache / apache/hudi

Optimize log block reading by removing seeks to check corrupted blocks

Open
#15,911 2 comments 0 reactions 1 assignee Claimed by @prashantwason View on GitHub
from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The code currently does an eager isCorruptedCheck for which we do a seek and then a read which invalidates our internal buffers in opened file stream to the log file and makes a call to DataNode to start a new blockReader.

The seek + read becomes apparent when we do cross datacenter reads or where the latency to the file is HIGH. In cases, a single RPC will cost us about 120ms + Cost of RPC (west coast to east coast) so this seek is bad for performance.

Delaying the corrupt check also gives us many benefits in low latency env where we see times reducing from (5 to 8 sec) to (3s to < 500ms) for a moderately sized files of 250MB.

NOTE:  The more number of log blocks to read, the greater the performance improvements.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6116
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.