cockroachdb / cockroachdb/cockroach
kvserver: getBatchRequestQPS breaks under extreme kv-batch-sizes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`getBatchRequestQPS` estimates the cost of a `BatchRequest`. In the general case, without an `AddSSTableRequest`, it estimates this cost as 1 query per second.
https://github.com/cockroachdb/cockroach/blob/001fbe82923805507a0e6d28431d0dd666a3cfee/pkg/kv/kvserver/replica_send.go#L1049-L1056
This estimation however, breaks down when `kv-batch-size` is set to an extreme value. For example, if `kv-batch-size` is set to an extremely high value relative to the default of `100000`, then the overhead of serializing gRPC calls is no longer the dominating factor and 1 QPS would be too high of an estimate.
The opposite is also true where if we set `kv-batch-size` to 1, the estimated QPS would be overcompensating for non-serialization costs and be too low.
The latter case can be observed/reproduced through the metamorphic constant `kv-batch-size` which can randomly be set to 1 if metamorphic constants are enabled:
https://github.com/cockroachdb/cockroach/blob/02b02804c676c96c9e2875ae19cb81394cc7db73/pkg/sql/row/kv_batch_fetcher.go#L55-L58
When this constant is set to 1 and we run a test that uses qps-based load threshold (i.e. `splits/load/spanning` roachtest), we can see that queries per second is much higher than the estimate, causing unexpected splits.
`(monitor.go:153).Wait: monitor failure: 6 splits, expected between 0 and 0 splits (ranges 7 initial 1)`
```
I231013 19:25:31.633429 315 13@kv/kvserver/store_rebalancer.go:803 ⋮ [T1,Vsystem,n1,s1,store-rebalancer,obj=‹qps›] 1344 transferring lease for r78 load=‹(queries-per-second=1310.2 cpu-per-second=855ms)› to store s2 load=‹(queries-per-second=7.5 cpu-per-second=22ms)› from local store s1 load=‹(queries-per-second=1563.2 cpu-per-second=958ms)›
```
Ideally `getBatchRequestQPS` would be able to account for these extreme cases.
Jira issue: CRDB-32537
Contributor guide
Assessment
This issue has not been assessed yet.