cockroachdb / cockroachdb/cockroach
kv: long running elastic workloads cause foreground workloads to stall
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Running a long running elastic workload can cause a foreground workload to stall after a period of time. This occurs with RACv2 in `apply_to_elastic` mode.
**To Reproduce**
To manually reproduce, start a fixed throughput workload with normal priority that uses ~50% of CPU. Observe stable throughput and latency. Then start an elastic workload for 30m with low priority. The elastic workload should be relatively low and fixed concurrency but enough that the cluster won't be able to process all of it. After ~15m the foreground workload will begin to be impacted.
To automatically reproduce run this command. In case the system change.
```
COCKROACH_RANDOM_SEED=5128980479178314991 roachtest run perturbation/metamorphic/elasticWorkload
```
This starts the test with the following variations:
```
test variations are: seed: 5128980479178314991,
fillDuration: 10m0s,
maxBlockBytes: 4096,
perturbationDuration: 30m0s,
validationDuration: 5m0s,
ratioOfMax: 0.500000,
splits: 10000,
numNodes: 12,
numWorkloadNodes: 1,
vcpu: 8,
disks: 1,
memory: high,
leaseType: epoch,
cloud: gce,
perturbation: {}
```
About 10-15 minutes into the perturbation the foreground workload becomes "unstable" and total throughput on the cluster drops well below the initial throughput without the elastic workload running. For periods of up to 5s there is no throughput on the cluster.
On at least one node we get high goroutine scheduling latency, and on three different nodes the WAL fsync latency jumps to 15s.
**Expected behavior**
AC should prevent the background workload from impacting the foreground workload.
**Additional data / screenshots**
* [A recent run](https://github.com/cockroachdb/cockroach/issues/134584#issue-2642208726)
* [Grafana link](https://grafana.testeng.crdb.io/d/crdb-home/crdb-home?orgId=1&var-DS_PROMETHEUS=Prometheus&var-test_run_id=teamcity-17632465&var-test_name=perturbation-metamorphic-elasticworkload&from=1731010692290&to=1731014252516)
**Environment:**
- CockroachDB version - 24.3 and master
Jira issue: CRDB-44194
Epic CRDB-42900
Contributor guide
Assessment
This issue has not been assessed yet.