ClickHouse / ClickHouse/ClickHouse

RFC: Allow to give higher priority to low-latency queries

Open
#106,405 0 comments 3 reactions 0 assignees View on GitHub
comp-scheduler performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse

### Describe the situation

Let's suppose we have a server with constant 100% CPU load. This could be normal when there is a script doing heavy batch queries in a loop, and as the script has natural backpressure, it does not overload the server - the load is sustained. At the same time, the server can also serve quick queries from interactive users. We want to give priority to these queries. While it is possible with settings, it is better if done automatically.

There is a trivial solution for that: if query already takes a long time, lower its priority proportionally - prioritizing newer requests and deprioritizing older ones.

This might sound too trivial, but it makes sense:
- if a query takes a long time, finishing it ASAP is less priority than for short queries;
- if a query takes a long time, there is a higher chance that it is suboptimal, and it's okay to penalize it;
- if a query takes a long time, there is less chance that someone waits for it interactively;

### Which ClickHouse versions are affected?

.

### How to reproduce

.

### Expected performance

Introduce new settings:

`os_threads_nice_value_query_increase_per_second` - every second of the query runtime, we will increase the nice value.

`priority_increase_per_second` - every second of the query runtime, we will increase the `priority` value (which is the roughest control on the query priorities, rarely used).

Introduce new configuration parameters for the workload scheduler:

`query_priority_increase_per_second` - every second of the query runtime, increase the value (lower the priority) accordingly.

`query_weight_lower_per_second` - every second of the query runtime, update the weight so it will equal to the original weight divided to the query runtime.

These configuration parameters apply for individual queries rather than to the whole workload, so it balances queries inside the workload (rather than different workloads).

### Related issues and pull requests

_No response_

### Additional context

The values of the settings can be floating point, e.g., 0.5 increase per second. For discrete controls like the OS nice value, it will mean increase by one every two seconds. The increase itself can be performed on shorter intervals rather than one second. We want to do it not more frequently than per one processed block or progress update and we don't want to do syscalls like sched_setpriority too frequently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.