cockroachdb / cockroachdb/pebble

internal/record: use direct I/O for the WAL and MANIFEST

Open
#1,159 6 comments 0 reactions 0 assignees View on GitHub
T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

WAL and MANIFEST might be good candidates for using direct I/O. The `LogWriter` already handles organizing writes into contiguous blocks. I'm not sure what impact, if any, on performance direct I/O would have.

I do think it would allow us to retry failed syncs during WAL and MANIFEST writes: [DB.Apply Fatalf](https://github.com/cockroachdb/pebble/blob/25f8258be7ea83e1736af7f48c4f691898ed1182/db.go#L585-L589), [logAndApply Fatalfs](https://github.com/cockroachdb/pebble/blob/25f8258be7ea83e1736af7f48c4f691898ed1182/version_set.go#L438-L449)

Currently, errors in these codepaths are fatal because `fsync`s of OS-buffered files cannot be retried. The OS marks errored buffers as clean, meaning a retried `fsync` will _not_ sync the buffer and the file's contents remain unchanged regardless of a retry. https://wiki.postgresql.org/wiki/Fsync_Errors

This was motivated by thinking about [@sumeerbhola's automated ballast file suggestion](https://groups.google.com/a/cockroachlabs.com/g/storage/c/E-_x0EvcoaY) for detecting out-of-disk conditions. It's a really nice solution. The one sticking point is that an `ENOSPC` may occur during `fsync`. I'm not sure under what conditions `ENOSPC` may surface from `fsync` rather than the preceding `write`, but I suspect it may happen when the filesystem needs to allocate new metadata blocks. I'm not sure but maybe on copy-on-write file systems all block allocations happen during `fsync`?

Jira issue: PEBBLE-211

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the LogWriter implementation and the fatal error paths in db.go and version_set.go. Investigate whether direct I/O is suitable for WAL and MANIFEST writes, including sync retry behavior and performance; done requires a decided approach supported by tests or measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.