cockroachdb / cockroachdb/pebble

db: prototype use of io_uring

Open
#4,684 5 comments 1 reaction 0 assignees View on GitHub
A-storage C-enhancement C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

io_uring is a Linux kernel system call API providing asynchronous I/O. Adopting io_uring has the potential to provide a few benefits:

1. Reduced CPU usage: io_uring reduces cpu usage through a) reducing the number of system calls that need to be performed, and b) avoiding memcpy-ing of data by sharing buffers between userland and kernel land.
2. Through adopting io_uring and altering our I/O APIs, we'll be able to also adopt `O_DIRECT` for writing to the write-ahead log and manifest. This should reduce CPU usage and have a slight improvement on write latency (see #1159).

### Existing I/O patterns

All production write I/O code paths perform sequential writes. The primary write paths are:

* sstables (sorted string tables)
* blob files
* WAL (write-ahead log)
* manifest

SSTables and blob files are the largest source of write I/O in the database. When writing an sstable or blob file, data is appended sequentially. When the entirety of the file has been written, it's fsynced to ensure all previously-appended data has been durably persisted. Data is also fsynced asynchronously in the background as the file is being written to smoothe I/O over time, avoiding a glut of dirty pages that need to be synced all at once when the file is complete. This smoothing is performed by [vfs.SyncingFile](https://github.com/cockroachdb/pebble/blob/master/vfs/syncing_file.go).

The write-ahead log (WAL) is the most latency-sensitive write path. All committing batches need to append their batch to the write-ahead log. Committing batches also typically (eg, when committing a CockroachDB SQL transaction) need to wait for their write to be durably synced. The throughput of the write-ahead log is less than the sstables and blob files. Writing to the write-ahead log is performed by the [record.LogWriter](https://github.com/cockroachdb/pebble/blob/master/record/log_writer.go).

The manfiest has a low volume of writes and is all sequential.

The vast majority of reads across Pebble are random block reads. These reads are performed by the [block.Reader](https://github.com/cockroachdb/pebble/blob/52551f8ad9c080fdacdd77fed924af22da673b0f/sstable/block/block.go#L397). The reading goroutine blocks until the read is complete.

### Asynchronous interface

One of the largest challeneges in adopting io_uring is its asynchronous interface and existing I/O call sites that expect synchronous I/O. Pebble has an existing [vfs.FS](https://pkg.go.dev/github.com/cockroachdb/pebble@v0.0.0-20250501235351-52551f8ad9c0/vfs#FS) interface used to abstract over the filesystem and I/O. This interface is designed for synchronous I/O. A call to `(vfs.File).ReadAt` or `(vfs.File).Write` blocks the goroutine until the operation is complete. An integration of io_uring should give careful consideration to the interface exposed to the rest of Pebble: is the asynchronicity exposed through the vfs interface, or is it fully contained behind the existing synchronous interface?

**Timing disk operations**

Pebble monitors the latency of write operations using [vfs.WithDiskHealthChecks](https://pkg.go.dev/github.com/cockroachdb/pebble@v0.0.0-20250501235351-52551f8ad9c0/vfs#WithDiskHealthChecks), relying on synchronous I/O.

**Encryption-at-rest**

CockroachDB implements encryption-at-rest through wrapping a `vfs.FS` with a an implementation that transparently performs encryption and decryption. The interface through which we expose io_uring needs to support the layering of encryption-at-rest.

https://kernel.dk/io_uring.pdf
https://unixism.net/loti/index.html
https://github.com/axboe/liburing
https://github.com/cockroachdb/pebble/blob/master/vfs/syncing_file.go
https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/
https://developers.mattermost.com/blog/hands-on-iouring-go/
https://github.com/pawelgaczynski/giouring

Jira issue: PEBBLE-437

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.