Add `fsync()` of _directories_ when renaming files to `.committed`
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 876
- Forks
- 260
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 157
Description
While reading about atomic rename semantics as part of #7029, we've surfaced something further that we may want to do to ensure the renames succeed, with the contents of the newly created file. Specifically, if we _don't_ `fsync()` the parent directory, it is technically possible for the rename (especially around restarts) to access a version of the directory metadata that doesn't know the file exists. So we should, when dealing with `.committed` files:
1) Before `close`ing a `.committed` file, `fsync()` that file
2) After `close()`, call `fsync()` on the parent directory, to minimise loss window, and be sure it's known before an incoming `rename()` call
3) After `rename()`, call `fsync()` on the destination directory, to minimise the window where the rename could be lost
This probably requires shifting to direct `fopen`/`fwrite` calls, so we have file-descriptor access for `fsync()`, and dropping the current use of `std::filesystem`/`std::ofstream` in many places.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the handling of `.committed` files and the current `std::filesystem`/`std::ofstream` write and rename paths. Trace the file close, parent-directory, and destination-directory lifecycle, then verify that each listed durability point is covered and that existing persistence behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100