Introduce buffering for streams in HUDI

Input and Output streams created in HUDI through calls to HoodieWrapperFileSystem do not include any buffering unless the underlying file system implements buffering.

DistributedFileSystem (over HDFS) does not implement any buffering. This leads to very large number of small-sized IO calls being send to the HDFS while performing HUDI IO operations like reading parquet, writing parquet, reading/writing log files, reading/writing instants, etc. 

This patch introduces buffering at the HoodieWrapperFileSystem level so that all types of reads and writes benefit from buffering.

 

In my tests with at scale on HDFS writing 1million records into a parquet file (read from an existing parquet file in the same dataset), I observed the following benefits:
# about 40% reduction in total time to run the test  
# Total write calls to HDFS reduced from 19.1M -> 328
# Total read calls reduced from 229M -> 515K

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1554
- Type: Improvement

---

## Comments

08/Aug/21 20:17;githubbot;hudi-bot edited a comment on pull request #2496:
URL: https://github.com/apache/hudi/pull/2496#issuecomment-869762023

## CI report:

* ba72d3ee9f569bc68f21d410e672378881c954b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=501)


Bot commands
The @flinkbot bot supports the following commands:

- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org
;;;

---

09/Aug/21 04:21;githubbot;hudi-bot edited a comment on pull request #2496:
URL: https://github.com/apache/hudi/pull/2496#issuecomment-869762023

## CI report:

* ba72d3ee9f569bc68f21d410e672378881c954b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=501)


Bot commands
@hudi-bot supports the following commands:

- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org
;;;

Start with the HoodieWrapperFileSystem entry point described in the issue and inspect pull request #2496 for the work already underway. Check how reads and writes for parquet, log files, and instants are created, and consider the change complete when these operations use buffering and reduce small HDFS IO calls.

Written by the indexing model from the issue text.