<fstream>: Poor fstream performance on large files
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 11.2k
- Forks
- 1.7k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 22
Description
Describe the bug
Reading large files is very slow using ifstream, mainly if it's in storage with low IOPS, making it unusable in VMs or reading files from a network share.
The main problem is that files are read in small 4KB chunks, regardless of the buffer size that you write it into. Capturing the syscalls with drstrace or sysinternals shows STL calling ReadFile repeatedly with 4KB reads, even when I pass it a GB sized buffer to write into. Looking into the source code I believe these are the problematic functions (though I didn't try fixing it myself): https://github.com/microsoft/STL/blob/b2f1556d2a0ad9a1ec0d764594972ce14c08a25c/tools/inc/stljobs.h#L379-L410 , which run with bufferSize = 4096.
This stackoverflow thread shows some benchmark numbers: https://stackoverflow.com/questions/32544389/c-sharp-file-readallbytes-vs-stdifstream-windows . This comment on onnxruntime shows some more: https://github.com/microsoft/onnxruntime/pull/4535#issuecomment-659737922
I'd like STL to use larger reads into my buffer when the buffer is large. Reading a GB sized file in 4KB reads kills I/O performance.
Command-line test case
#include <fstream>
#include <vector>
int main() {
std::ifstream s(R"(Path to a large file in your machine)", std::ifstream::binary);
std::vector<char> buffer(60701790);
s.read(buffer.data(), buffer.size());
return 0;
}
Expected behavior
Performance is good/improves with larger buffers.
Actual behavior
The stackoverflow thread shows some numbers:
- OP says reading a MBs sized file from a network share takes 3 times longer than in C#
- Another response, with multiple 37MB files, shows increasing performance using the Windows API or C# with larger buffers, while the STL plateaus at 4KB:
+------------+---------+----------+
| Block_Size | SLT | Kernel32 |
+------------+---------+----------+
| 1KB | 976 | 1,101 |
| 4KB | 1,027 | 1,011 |
| 32KB | 969 | 768 |
| 1MB | 981 | 530 |
| 5MB | 1,008 | 541 |
+------------+---------+----------+
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tools/inc/stljobs.h at lines 379-410 and inspect how the 4096-byte bufferSize drives reads for ifstream. Reproduce the issue with the provided large-file command-line test on Windows, including a network share or low-IOPS storage if available. Done means demonstrating improved performance with a larger destination buffer without changing the expected stream behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100