HarperFast / HarperFast/rocksdb-js
Transaction Log: Investigate using `WriteFileGather()` on Windows
- Dominant language
- C++
- Stars
- 21
- Forks
- 2
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
The current transaction log system writes each entry serially and makes several system calls.
Windows has a function called `WriteFileGather()` which is similar to the posix `writev()` function which writes a bunch of data buffers in a single system call.
There are a bunch of caveats with using this function that make implementing it complicated including:
* Alignment Requirements
* Buffer addresses must be aligned on system page boundaries (typically 4KB on x86/x64)
* File offsets must also be page-aligned
* File Handle Requirements
* The file must be opened with `FILE_FLAG_OVERLAPPED` and `FILE_FLAG_NO_BUFFERING`
* `FILE_FLAG_NO_BUFFERING` imposes additional constraints: transfer sizes must be multiples of the disk sector size, and file positions must be sector-aligned
* Synchronization Complexity
* Always operates asynchronously, even if the handle wasn't opened for overlapped I/O
* You must use the OVERLAPPED structure and wait for completion via event signaling or I/O completion ports
* Cannot rely on the function completing synchronously
* Buffer Management
* The `FILE_SEGMENT_ELEMENT` array and its buffers must remain valid until the operation completes
* Premature deallocation or modification can cause data corruption or crashes
* Limited Error Reporting
* Partial writes aren't clearly reported - you may need to track which segments succeeded
* Error handling is more complex than simple `WriteFile()`
At the end of the day, the overhead of meeting alignment requirements is best suited for large, page-aligned transfers and may negate benefits for small I/O operations.
Contributor guide
Assessment
This issue has not been assessed yet.