cockroachdb / cockroachdb/cockroach

kvserver: getBatchRequestQPS breaks under extreme kv-batch-sizes

Open
#112,664 0 comments 0 reactions 0 assignees View on GitHub
A-kv-server C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`getBatchRequestQPS` estimates the cost of a `BatchRequest`. In the general case, without an `AddSSTableRequest`, it estimates this cost as 1 query per second.

https://github.com/cockroachdb/cockroach/blob/001fbe82923805507a0e6d28431d0dd666a3cfee/pkg/kv/kvserver/replica_send.go#L1049-L1056

This estimation however, breaks down when `kv-batch-size` is set to an extreme value. For example, if `kv-batch-size` is set to an extremely high value relative to the default of `100000`, then the overhead of serializing gRPC calls is no longer the dominating factor and 1 QPS would be too high of an estimate.

The opposite is also true where if we set `kv-batch-size` to 1, the estimated QPS would be overcompensating for non-serialization costs and be too low.

The latter case can be observed/reproduced through the metamorphic constant `kv-batch-size` which can randomly be set to 1 if metamorphic constants are enabled:

https://github.com/cockroachdb/cockroach/blob/02b02804c676c96c9e2875ae19cb81394cc7db73/pkg/sql/row/kv_batch_fetcher.go#L55-L58

When this constant is set to 1 and we run a test that uses qps-based load threshold (i.e. `splits/load/spanning` roachtest), we can see that queries per second is much higher than the estimate, causing unexpected splits.

`(monitor.go:153).Wait: monitor failure: 6 splits, expected between 0 and 0 splits (ranges 7 initial 1)`

```
I231013 19:25:31.633429 315 13@kv/kvserver/store_rebalancer.go:803 ⋮ [T1,Vsystem,n1,s1,store-rebalancer,obj=‹qps›] 1344 transferring lease for r78 load=‹(queries-per-second=1310.2 cpu-per-second=855ms)› to store s2 load=‹(queries-per-second=7.5 cpu-per-second=22ms)› from local store s1 load=‹(queries-per-second=1563.2 cpu-per-second=958ms)›
```

Ideally `getBatchRequestQPS` would be able to account for these extreme cases.

Jira issue: CRDB-32537

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.