clockworklabs / clockworklabs/SpacetimeDB
Reimplement: commitlog: Direct I/O
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 25.2k
- Forks
- 1.1k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 46
Description
This issue tracks reimplementation of the work from stale PR #1079, which is being closed because it is too out of date to merge directly.
Original PR: https://github.com/clockworklabs/SpacetimeDB/pull/1079
Original author: @kim
Original branch: `kim/commitlog2/direct-io`
Base branch: `master`
## Original PR summary
Introduce page-aligned buffered reads + writes. This is required for direct I/O (O_DIRECT or equivalent), which is now the default.
Direct I/O essentially means to bypass the OS's page cache and operate directly on the device. It does not mean that data is automatically more durable once written, fsync is still required. Optionally, the patch allows to enable O_DSYNC, which blocks a write until the equivalent of fdatasync has occurred.
Both options likely have a performance impact, which needs to be evaluated. There also some performance optimizations of the presented implemention possible, e.g. re-using buffer allocations or scatter/gather ("vectored") I/O.
Stacked on top of #985
Expected complexity level and risk
4
Testing
Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!
-
Run test suite under Linux
-
Check that the code compiles under occult operating systems (macOS / Windows)
-
Check that the code works under occult operating systems.
Follow-up
- Reimplement this change in a fresh PR against current
master. - Carry forward any still-relevant context from the original PR discussion and review.
- Link the new implementation PR back to the original stale PR for historical context.
- Reimplement this change in a fresh PR against current
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read stale PR #1079 and the stacked issue #985, then compare their still-relevant context with current master. Run the test suite under Linux and check compilation and behavior on macOS and Windows; done means a fresh PR reimplements page-aligned buffered I/O for direct I/O and links back to the original PR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100