HarperFast / HarperFast/rocksdb-js

Transaction Log: Investigate using `WriteFileGather()` on Windows

Open
#220 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

The current transaction log system writes each entry serially and makes several system calls.

Windows has a function called `WriteFileGather()` which is similar to the posix `writev()` function which writes a bunch of data buffers in a single system call.

There are a bunch of caveats with using this function that make implementing it complicated including:
* Alignment Requirements
* Buffer addresses must be aligned on system page boundaries (typically 4KB on x86/x64)
* File offsets must also be page-aligned
* File Handle Requirements
* The file must be opened with `FILE_FLAG_OVERLAPPED` and `FILE_FLAG_NO_BUFFERING`
* `FILE_FLAG_NO_BUFFERING` imposes additional constraints: transfer sizes must be multiples of the disk sector size, and file positions must be sector-aligned
* Synchronization Complexity
* Always operates asynchronously, even if the handle wasn't opened for overlapped I/O
* You must use the OVERLAPPED structure and wait for completion via event signaling or I/O completion ports
* Cannot rely on the function completing synchronously
* Buffer Management
* The `FILE_SEGMENT_ELEMENT` array and its buffers must remain valid until the operation completes
* Premature deallocation or modification can cause data corruption or crashes
* Limited Error Reporting
* Partial writes aren't clearly reported - you may need to track which segments succeeded
* Error handling is more complex than simple `WriteFile()`

At the end of the day, the overhead of meeting alignment requirements is best suited for large, page-aligned transfers and may negate benefits for small I/O operations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.