ClickHouse / ClickHouse/ClickHouse

max_local_read_bandwidth_for_server throttles reads served from the OS page cache

Open
#118,463 0 comments 0 reactions 0 assignees View on GitHub
comp-scheduler external st-accepted unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

Nebius

### Describe the unexpected behaviour

`max_local_read_bandwidth_for_server` (and the per-query `max_local_read_bandwidth`) throttles bytes that are served from the OS page cache exactly like bytes that hit the disk. A query whose data is fully cached in RAM is slowed down to the configured disk bandwidth, even though it produces almost no block-device I/O.

The setting is documented as "The maximum speed of local reads in bytes per second" and was introduced in #48242 to protect running queries from BACKUP/other IO. Its natural use is "do not let ClickHouse exceed what the underlying disk can deliver". Counting page-cache hits makes it useless for that purpose: the limit has to be set to the disk bandwidth or above (and then it protects nothing), or below it (and then it penalizes cached reads that never touch the disk).

### Which ClickHouse versions are affected?

26.6.2.160; the code is unchanged on current master.

### How to reproduce

* ClickHouse server version: 26.6.2.160 (also present on current master, see code references below)
* `max_local_read_bandwidth_for_server = 350000000` (350 MB/s), disks are network SSD with a 1 GiB/s read limit
* default `local_filesystem_read_method = pread_threadpool`
* Run a SELECT that scans ~3 GiB of compressed data per node twice, so that the second run is served from the page cache.

Observed in `system.query_log` for the cached run, per node (7 shards, numbers are almost identical on each):

| ProfileEvent | value |
|---|---|
| `LocalReadThrottlerBytes` | 2.84–2.88 GiB |
| `ThreadPoolReaderPageCacheHitBytes` | 2.69–2.80 GiB (~97% of the throttled bytes) |
| `ThreadPoolReaderPageCacheMissBytes` | 76–158 MiB |
| `LocalReadThrottlerSleepMicroseconds` | 93–115 s (summed over threads) |
| `query_duration_ms` | 17.5–19.2 s |

Raising `max_local_read_bandwidth_for_server` to 850000000 brought the same query down to 3.5–10 s; throttler sleep dropped to 0–8 s and per-node read throughput went from ~330 MB/s (the old cap) to ~860 MB/s (the new cap). The disk itself never came close to its limit in either run (peak ~100 MiB/s per volume as reported by the storage system).

### Expected behavior

Reads served from the OS page cache should not consume tokens of the local read throttler (or at least there should be a way to opt out of counting them). A minimal change would be to add a `from_page_cache` flag to `IAsynchronousReader::Result`, set it in `ThreadPoolReader` on the `RWF_NOWAIT` success path, and skip `throttler->throttle()` in `AsynchronousReadBufferFromFileDescriptor::readImpl` when it is set. The synchronous `pread` path cannot know whether the data was cached, but with the default `pread_threadpool` method this would already cover the common case.

### Error message and/or stacktrace

_No response_

### Related issues and pull requests

#48242 introduced the setting.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with IAsynchronousReader::Result and trace the RWF_NOWAIT success path in ThreadPoolReader, then inspect throttling in AsynchronousReadBufferFromFileDescriptor::readImpl. Ensure page-cache reads do not consume local-read throttler tokens while preserving the synchronous pread behavior; verify the reported throttler and page-cache metrics for cached reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
73/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.