ClickHouse / ClickHouse/silk

Potential `io_uring` optimizations, experiments and improvements.

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
325
Forks
13
Avg merge
2d 18h
Merged PRs (30d)
23

Description

Silk uses [io_uring as ground truth](https://clickhouse.com/blog/silk#user-content-tldr). However the current state of silk uses `io_uring` without much optimizations. I feel like there are some opportunities.

These optimizations are heavily inspired by the paper [io_uring for High-Performance DBMSs: When and How to Use It](https://arxiv.org/pdf/2512.04859) and bit of playing with `io_uring` raw apis.

Would like to use this issue to track such optimizations, improvements, experiments and discussions related to it.

Following TODOs are just here to experiment with, no guarantee it will end up in silk. Also these TODOs can be added/removed/updated as I experiment with.

### TODOs

- [x] [Fixed buffers instead of per-io buffer allocation.](https://github.com/ClickHouse/silk/pull/72)
- [x] [PMC counters support (at least basic counters like `cycles`, `instructions` and `context-switches`); This will be used to measure the impact of io_uring optimizations](https://github.com/ClickHouse/silk/pull/93)
- [ ] Register files. Avoid file descriptors sharing contention between different threads of same process.
- [ ] Register Ring file descriptors. Blocked by `submissionLock`.
- [ ] SINGLE_ISSUER, DEFER_TASKRUN, COOP_TASKRUN flags. Blocked by `submissionLock`
- [ ] Direct access to NVMe queues. Bypassing the entire kernel storage IO stack (`OP_URING_CMD` opcode)
- [ ] Visibility on kernel `io_worker` fallback (frequent fallback to `io_worker` may signal wrong IO pattern)
- [ ] IOSQE_CQE_SKIP_SUCCESS on internal SQEs (currently [Silk submits SQEs whose successful CQE is intentionally discarded](https://github.com/ClickHouse/silk/blob/092ad05978c3f034bd31e2a476ecde83daa2b9e7/src/fibers/fiber.cpp?plain=1#L1805-L1809))
- [ ] Multishot accept. Silk [currently does two scheduler round-trips per accepted connection (`poll()` + `accept4`)](https://github.com/ClickHouse/silk/blob/3b14bc854042a271bda8a09251adb0ab202889fe/src/perf/fiber-http.cpp?plain=1#L206-L219). And can be replaced with single. One persistent multishot SQE delivers a CQE per connection with the accepted fd carried inline.

### Blockers
In silk, thread-mode fibers and proxy fibers can call read/write/poll from a thread that is not the target CPU's scheduler thread. Silk has [submissionLock](https://github.com/ClickHouse/silk/blob/7034a52bd7dc2eda39cef20e6071f3e95e95c039/src/fibers/fiber.cpp?plain=1#L565-L568) to have synchronization between these different actors. This is one of the single biggest blocker to use io_uring's SINGLE_ISSUER optimization.

Which will enable us to use [other optimizations](https://man7.org/linux/man-pages/man2/io_uring_enter2.2.html) like `DEFER_TASKRUN` which enables kernel to run `task_run` (an task that actually moves completed IO to CQE) only during `io_uring_syscall_enter` instead of interrupting the user-space randomly. Expectation is to have more predictable IO performance (less variance in p99 tail latency)

SINGLE_ISSUER also enables us to do [io_uring_register_ring_fd optimization](https://man7.org/linux/man-pages/man3/io_uring_register_ring_fd.3.html) which reduces the per-syscall overhead of grabbing ring fd reference every single time `io_uring_enter` syscall is invoked.

One workaround I can think of is to use `submissionInbox` model (similar to [how silk already uses cancelQueue and sleepQueue](https://github.com/ClickHouse/silk/blob/7034a52bd7dc2eda39cef20e6071f3e95e95c039/src/fibers/fiber.cpp?plain=1#L607-L608)) where still multiple threads can submit the IO, but they submit to the inbox and one actor per ring (per CPU) actually submits the IO via `io_uring_enter` syscall without needing any synchronization. But I don't how this will impact overall latency and throughput. Need to be measured before adding this.

### Methodology
1. Every optimizations have to be evaluated using PMC counters (direct impact)
2. Perf benchmarks (file-perf, net-perf, etc) - For end-to-end throughput and latency gains.
3. Clear documentation about the optimizations, whether it targets CPU bound or IO bound workloads.

Contributor guide

Open the contributing guide

Research direction

Start by reading the io_uring submission and synchronization paths in src/fibers/fiber.cpp, especially submissionLock, cancelQueue, and sleepQueue. Review the linked scheduler and accept paths in src/perf/fiber-http.cpp, then run the relevant file-perf or net-perf benchmark with PMC counters. Done means one selected optimization is implemented, measured for throughput and latency, and documented with its workload scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.