cockroachdb / cockroachdb/cockroach
kv: latency spike after tpcc workload ramps up
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
An issue was observed on the CCT cluster on the tpcc_drop workload. The workload loops through the following steps:
1. Renames any old tpcc database if one exists
2. Loads a new tpcc workload with 3000 warehouses of data
3. Drops the renamed workload
4. Runs tpcc for 60 minutes
We've found that when the cluster was running on 23.1, every hour we'd see one latency spike as the IMPORT statements are run in step 2. After finalizing on 23.2, we're observing a second latency spike (as well as a throughput drop), approximately 5 minutes after the IMPORT and drop are complete and the tpcc workload has started ramping up.
23.1 (latency spikes confined to IMPORT window)
23.2 finalized (latency spikes occur _after_ IMPORT is complete and workload is ramping up):
This is being discussed further [here](https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1701013647475009).
Jira issue: CRDB-34031
Contributor guide
Assessment
This issue has not been assessed yet.