cockroachdb / cockroachdb/pebble
internal/record: use direct I/O for the WAL and MANIFEST
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
WAL and MANIFEST might be good candidates for using direct I/O. The `LogWriter` already handles organizing writes into contiguous blocks. I'm not sure what impact, if any, on performance direct I/O would have.
I do think it would allow us to retry failed syncs during WAL and MANIFEST writes: [DB.Apply Fatalf](https://github.com/cockroachdb/pebble/blob/25f8258be7ea83e1736af7f48c4f691898ed1182/db.go#L585-L589), [logAndApply Fatalfs](https://github.com/cockroachdb/pebble/blob/25f8258be7ea83e1736af7f48c4f691898ed1182/version_set.go#L438-L449)
Currently, errors in these codepaths are fatal because `fsync`s of OS-buffered files cannot be retried. The OS marks errored buffers as clean, meaning a retried `fsync` will _not_ sync the buffer and the file's contents remain unchanged regardless of a retry. https://wiki.postgresql.org/wiki/Fsync_Errors
This was motivated by thinking about [@sumeerbhola's automated ballast file suggestion](https://groups.google.com/a/cockroachlabs.com/g/storage/c/E-_x0EvcoaY) for detecting out-of-disk conditions. It's a really nice solution. The one sticking point is that an `ENOSPC` may occur during `fsync`. I'm not sure under what conditions `ENOSPC` may surface from `fsync` rather than the preceding `write`, but I suspect it may happen when the filesystem needs to allocate new metadata blocks. I'm not sure but maybe on copy-on-write file systems all block allocations happen during `fsync`?
Jira issue: PEBBLE-211
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the LogWriter implementation and the fatal error paths in db.go and version_set.go. Investigate whether direct I/O is suitable for WAL and MANIFEST writes, including sync retry behavior and performance; done requires a decided approach supported by tests or measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100