cockroachdb / cockroachdb/pebble
db: prototype use of io_uring
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
io_uring is a Linux kernel system call API providing asynchronous I/O. Adopting io_uring has the potential to provide a few benefits:
1. Reduced CPU usage: io_uring reduces cpu usage through a) reducing the number of system calls that need to be performed, and b) avoiding memcpy-ing of data by sharing buffers between userland and kernel land.
2. Through adopting io_uring and altering our I/O APIs, we'll be able to also adopt `O_DIRECT` for writing to the write-ahead log and manifest. This should reduce CPU usage and have a slight improvement on write latency (see #1159).
### Existing I/O patterns
All production write I/O code paths perform sequential writes. The primary write paths are:
* sstables (sorted string tables)
* blob files
* WAL (write-ahead log)
* manifest
SSTables and blob files are the largest source of write I/O in the database. When writing an sstable or blob file, data is appended sequentially. When the entirety of the file has been written, it's fsynced to ensure all previously-appended data has been durably persisted. Data is also fsynced asynchronously in the background as the file is being written to smoothe I/O over time, avoiding a glut of dirty pages that need to be synced all at once when the file is complete. This smoothing is performed by [vfs.SyncingFile](https://github.com/cockroachdb/pebble/blob/master/vfs/syncing_file.go).
The write-ahead log (WAL) is the most latency-sensitive write path. All committing batches need to append their batch to the write-ahead log. Committing batches also typically (eg, when committing a CockroachDB SQL transaction) need to wait for their write to be durably synced. The throughput of the write-ahead log is less than the sstables and blob files. Writing to the write-ahead log is performed by the [record.LogWriter](https://github.com/cockroachdb/pebble/blob/master/record/log_writer.go).
The manfiest has a low volume of writes and is all sequential.
The vast majority of reads across Pebble are random block reads. These reads are performed by the [block.Reader](https://github.com/cockroachdb/pebble/blob/52551f8ad9c080fdacdd77fed924af22da673b0f/sstable/block/block.go#L397). The reading goroutine blocks until the read is complete.
### Asynchronous interface
One of the largest challeneges in adopting io_uring is its asynchronous interface and existing I/O call sites that expect synchronous I/O. Pebble has an existing [vfs.FS](https://pkg.go.dev/github.com/cockroachdb/pebble@v0.0.0-20250501235351-52551f8ad9c0/vfs#FS) interface used to abstract over the filesystem and I/O. This interface is designed for synchronous I/O. A call to `(vfs.File).ReadAt` or `(vfs.File).Write` blocks the goroutine until the operation is complete. An integration of io_uring should give careful consideration to the interface exposed to the rest of Pebble: is the asynchronicity exposed through the vfs interface, or is it fully contained behind the existing synchronous interface?
**Timing disk operations**
Pebble monitors the latency of write operations using [vfs.WithDiskHealthChecks](https://pkg.go.dev/github.com/cockroachdb/pebble@v0.0.0-20250501235351-52551f8ad9c0/vfs#WithDiskHealthChecks), relying on synchronous I/O.
**Encryption-at-rest**
CockroachDB implements encryption-at-rest through wrapping a `vfs.FS` with a an implementation that transparently performs encryption and decryption. The interface through which we expose io_uring needs to support the layering of encryption-at-rest.
https://kernel.dk/io_uring.pdf
https://unixism.net/loti/index.html
https://github.com/axboe/liburing
https://github.com/cockroachdb/pebble/blob/master/vfs/syncing_file.go
https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/
https://developers.mattermost.com/blog/hands-on-iouring-go/
https://github.com/pawelgaczynski/giouring
Jira issue: PEBBLE-437
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.