ClickHouse / ClickHouse/ClickHouse
max_local_read_bandwidth_for_server throttles reads served from the OS page cache
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
Nebius
### Describe the unexpected behaviour
`max_local_read_bandwidth_for_server` (and the per-query `max_local_read_bandwidth`) throttles bytes that are served from the OS page cache exactly like bytes that hit the disk. A query whose data is fully cached in RAM is slowed down to the configured disk bandwidth, even though it produces almost no block-device I/O.
The setting is documented as "The maximum speed of local reads in bytes per second" and was introduced in #48242 to protect running queries from BACKUP/other IO. Its natural use is "do not let ClickHouse exceed what the underlying disk can deliver". Counting page-cache hits makes it useless for that purpose: the limit has to be set to the disk bandwidth or above (and then it protects nothing), or below it (and then it penalizes cached reads that never touch the disk).
### Which ClickHouse versions are affected?
26.6.2.160; the code is unchanged on current master.
### How to reproduce
* ClickHouse server version: 26.6.2.160 (also present on current master, see code references below)
* `max_local_read_bandwidth_for_server = 350000000` (350 MB/s), disks are network SSD with a 1 GiB/s read limit
* default `local_filesystem_read_method = pread_threadpool`
* Run a SELECT that scans ~3 GiB of compressed data per node twice, so that the second run is served from the page cache.
Observed in `system.query_log` for the cached run, per node (7 shards, numbers are almost identical on each):
| ProfileEvent | value |
|---|---|
| `LocalReadThrottlerBytes` | 2.84–2.88 GiB |
| `ThreadPoolReaderPageCacheHitBytes` | 2.69–2.80 GiB (~97% of the throttled bytes) |
| `ThreadPoolReaderPageCacheMissBytes` | 76–158 MiB |
| `LocalReadThrottlerSleepMicroseconds` | 93–115 s (summed over threads) |
| `query_duration_ms` | 17.5–19.2 s |
Raising `max_local_read_bandwidth_for_server` to 850000000 brought the same query down to 3.5–10 s; throttler sleep dropped to 0–8 s and per-node read throughput went from ~330 MB/s (the old cap) to ~860 MB/s (the new cap). The disk itself never came close to its limit in either run (peak ~100 MiB/s per volume as reported by the storage system).
### Expected behavior
Reads served from the OS page cache should not consume tokens of the local read throttler (or at least there should be a way to opt out of counting them). A minimal change would be to add a `from_page_cache` flag to `IAsynchronousReader::Result`, set it in `ThreadPoolReader` on the `RWF_NOWAIT` success path, and skip `throttler->throttle()` in `AsynchronousReadBufferFromFileDescriptor::readImpl` when it is set. The synchronous `pread` path cannot know whether the data was cached, but with the default `pread_threadpool` method this would already cover the common case.
### Error message and/or stacktrace
_No response_
### Related issues and pull requests
#48242 introduced the setting.
### Additional context
_No response_
Contributor guide
Research direction
Start with IAsynchronousReader::Result and trace the RWF_NOWAIT success path in ThreadPoolReader, then inspect throttling in AsynchronousReadBufferFromFileDescriptor::readImpl. Ensure page-cache reads do not consume local-read throttler tokens while preserving the synchronous pread behavior; verify the reported throttler and page-cache metrics for cached reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 73/100