cockroachdb / cockroachdb/cockroach

kv: latency spike after tpcc workload ramps up

Open
#115,431 9 comments 0 reactions 0 assignees View on GitHub
A-kv branch-release-23.2 C-bug O-testcluster P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

An issue was observed on the CCT cluster on the tpcc_drop workload. The workload loops through the following steps:

1. Renames any old tpcc database if one exists
2. Loads a new tpcc workload with 3000 warehouses of data
3. Drops the renamed workload
4. Runs tpcc for 60 minutes

We've found that when the cluster was running on 23.1, every hour we'd see one latency spike as the IMPORT statements are run in step 2. After finalizing on 23.2, we're observing a second latency spike (as well as a throughput drop), approximately 5 minutes after the IMPORT and drop are complete and the tpcc workload has started ramping up.

23.1 (latency spikes confined to IMPORT window)

image

23.2 finalized (latency spikes occur _after_ IMPORT is complete and workload is ramping up):

image

This is being discussed further [here](https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1701013647475009).

Jira issue: CRDB-34031

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.