microsoft / microsoft/STL

<fstream>: Poor fstream performance on large files

Open
#1,107 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
C++
Stars
11.2k
Forks
1.7k
Avg merge
4d 15h
Merged PRs (30d)
22

Description

Describe the bug

Reading large files is very slow using ifstream, mainly if it's in storage with low IOPS, making it unusable in VMs or reading files from a network share.

The main problem is that files are read in small 4KB chunks, regardless of the buffer size that you write it into. Capturing the syscalls with drstrace or sysinternals shows STL calling ReadFile repeatedly with 4KB reads, even when I pass it a GB sized buffer to write into. Looking into the source code I believe these are the problematic functions (though I didn't try fixing it myself): https://github.com/microsoft/STL/blob/b2f1556d2a0ad9a1ec0d764594972ce14c08a25c/tools/inc/stljobs.h#L379-L410 , which run with bufferSize = 4096.

This stackoverflow thread shows some benchmark numbers: https://stackoverflow.com/questions/32544389/c-sharp-file-readallbytes-vs-stdifstream-windows . This comment on onnxruntime shows some more: https://github.com/microsoft/onnxruntime/pull/4535#issuecomment-659737922

I'd like STL to use larger reads into my buffer when the buffer is large. Reading a GB sized file in 4KB reads kills I/O performance.

Command-line test case

#include <fstream>
#include <vector>

int main() {
    std::ifstream s(R"(Path to a large file in your machine)", std::ifstream::binary);
    std::vector<char> buffer(60701790);
    s.read(buffer.data(), buffer.size());
    return 0;
}

Expected behavior
Performance is good/improves with larger buffers.

Actual behavior
The stackoverflow thread shows some numbers:

  • OP says reading a MBs sized file from a network share takes 3 times longer than in C#
  • Another response, with multiple 37MB files, shows increasing performance using the Windows API or C# with larger buffers, while the STL plateaus at 4KB:
+------------+---------+----------+
| Block_Size |   SLT   | Kernel32 |
+------------+---------+----------+
| 1KB        |  976    |  1,101   |
| 4KB        |  1,027  |  1,011   |
| 32KB       |  969    |  768     |
| 1MB        |  981    |  530     |
| 5MB        |  1,008  |  541     |
+------------+---------+----------+

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/inc/stljobs.h at lines 379-410 and inspect how the 4096-byte bufferSize drives reads for ifstream. Reproduce the issue with the provided large-file command-line test on Windows, including a network share or low-IOPS storage if available. Done means demonstrating improved performance with a larger destination buffer without changing the expected stream behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.