cockroachdb / cockroachdb/cockroach

kv: QPS falls over time when batches are in use

Open
#122,845 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

A workload that performs batched reads on `23.2.4` gradually becomes less efficient, dropping from 1000 QPS to 750 QPS over the course of a 5 minute run. Batched reads are used in many workloads, most often as part of lookup joins. As load-based splits cause the number of ranges to increase for a given table, the performance of batched reads drops.

**To Reproduce**
```
export EMAIL=

# Start 3 node 23.2.4 cluster using Roachprod.
roachprod create $EMAIL-repro -n 3 --gce-machine-type=n2-standard-4 --local-ssd=false --geo --gce-zones='us-east1-b,us-east1-c,us-east1-d'
roachprod stage $EMAIL-repro release v23.2.4
roachprod start --secure $EMAIL-repro

# Create workload machine using Roachprod.
roachprod create $EMAIL-repro-load -n 1 --gce-machine-type=n2-standard-4 --local-ssd=false --geo --gce-zones='us-east1-b'
roachprod stage $EMAIL-repro-load release v23.2.4
roachprod get $EMAIL-repro:1 certs ./remote-certs-dir
roachprod put $EMAIL-repro-load ./remote-certs-dir ./certs
roachprod run $EMAIL-repro-load 'sudo chmod 600 ./certs/*'
rm -R ./remote-certs-dir

# Run HAProxy on the workload machine, to distribute load over cluster.
export crdb_ip=$(roachprod ip $EMAIL-repro:1 --external)
roachprod run $EMAIL-repro-load -- "./cockroach gen haproxy --certs-dir=./certs --host=$crdb_ip:26257"
roachprod run $EMAIL-repro-load -- 'sudo apt-get install -y haproxy'
roachprod run $EMAIL-repro-load -- 'haproxy -f haproxy.cfg > haproxy.log 2>&1 < /dev/null &'

# Run KV0 (write-only) workload for 5 minutes. This will trigger splits, resulting in 2 total ranges for the "kv" table.
roachprod run $EMAIL-repro-load -- './cockroach workload init kv "postgresql://root@localhost:26257/kv?sslcert=./certs/client.root.crt&sslkey=./certs/client.root.key&sslmode=verify-full&sslrootcert=./certs/ca.crt"'

roachprod run $EMAIL-repro-load -- './cockroach workload run kv --read-percent=0 --concurrency=40 --seed 1 --max-rate=5000 --duration=5m "postgresql://root@localhost:26257/kv?sslcert=./certs/client.root.crt&sslkey=./certs/client.root.key&sslmode=verify-full&sslrootcert=./certs/ca.crt"'

# Run KV100 (read-only with batch=100) workload for 5 minutes.
roachprod run $EMAIL-repro-load -- './cockroach workload run kv --batch=200 --write-seq=R1000000 --read-percent=100 --concurrency=40 --seed 1 --max-rate=1000 --duration=5m "postgresql://root@localhost:26257/kv?sslcert=./certs/client.root.crt&sslkey=./certs/client.root.key&sslmode=verify-full&sslrootcert=./certs/ca.crt"'
```

_EXPECTED_: QPS should start at 1000 and remain at 1000 throughout the duration of the run.

_ACTUAL_: QPS does start at 1000, but drops to 750 over the 5 minute run.

**Additional data / screenshots**

After the KV0 workload, there are 2 ranges for the `kv` table:
```
roachprod sql $EMAIL-repro-load --secure -- --database=kv -e "select count(*) from [show ranges from table kv]"
count
---------
2
```

The KV100 workload causes a feedback loop that gradually decreases QPS:

1. Increased CPU usage triggers load-based splits.
2. Adding ranges causes CRDB to use more CPU to perform the batched reads, even though the number of nodes and amount of data stays constant.
3. Go to step 1.

After 5 minutes of KV100, there are ~40 ranges, CPU usage has gradually risen to close to 100%, and QPS has dropped from 1000 to 750.
```
roachprod sql $EMAIL-repro-load --secure -- --database=kv -e "select count(*) from [show ranges from table kv]"
count
---------
38
```

Screenshot 2024-04-22 at 1 04 35 PM

Screenshot 2024-04-22 at 1 05 26 PM

Ideally, we'd find a batching algorithm that uses CPU proportional to the # nodes rather proportional to the # ranges. Then, it wouldn't matter as much if we performed additional range splits. Barring that, perhaps we should at least avoid splitting a single table into so many ranges (~40) when the cluster consists of only 3 nodes.

Jira issue: CRDB-38085

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.