Review HDFS file read performance enhancements
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 15
Description
I came across https://issues.apache.org/jira/browse/HADOOP-18287 which references some new APIs:
[High Performance Vectored Read API](https://issues.apache.org/jira/browse/HADOOP-11867)
[openFile](https://issues.apache.org/jira/browse/HADOOP-16202)
[ByteBufferPositionedReadable](https://hadoop.apache.org/docs/r2.10.0/api/org/apache/hadoop/fs/ByteBufferPositionedReadable.html)
There is also https://issues.apache.org/jira/browse/HADOOP-18287 which will enable these new features as they become available.
Contributor guide
Research direction
Start by reviewing HADOOP-18287 and its linked HADOOP-11867 and HADOOP-16202 issues, along with the ByteBufferPositionedReadable API. Trace Accumulo's current HDFS file-read path and document which APIs are available and what changes would be needed to use them; done means the performance enhancements have been evaluated with a clear implementation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100