cockroachdb / cockroachdb/cockroach

kvserver: add a roachtest repro for https://github.com/cockroachdb/cockroach/issues/160011

Open
#161,433 1 comment 0 reactions 0 assignees View on GitHub
A-kv branch-master branch-release-25.4 branch-release-26.1 C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

We should add a roachtest that repros what we saw in https://github.com/cockroachdb/cockroach/issues/160011. It should run 20k warehouses, tpcc inserts, 3 nodes.

- Claude said the closest roachtest we have is tpcc/published/headroom/25k) which runs tpcc, 20k warehouses, 3 nodes, 30 mins, 500 active workers.
- Find the relevant env files in https://cockroachlabs.slack.com/archives/C048HDZJSAY/p1768595332417589. The exact repro is:
```
git clone https://github.com/cockroachlabs/perf-efficiency-team
// use the given env files for cluster and workload configuration
./rpsak.sh create-only radu-tpcc
./rpsak.sh repeated radu-tpcc
```

- It should do be similar to below:
```
cd $crl

# Variables
CLUSTER=wenyicockroachlabscom-radurepro
NODES=3
WORKLOAD_NODE=4
ZONE=us-east1-d
MACHINE=n2-standard-32
SSDS=4
WAREHOUSES=20000
WORKERS=500
DURATION=30m

# 1. Create cluster
roachprod create $CLUSTER \
--nodes 4 \
--clouds gce \
--gce-machine-type $MACHINE \
--gce-zones $ZONE \
--gce-local-ssd-count $SSDS \
--username $(whoami) \
--lifetime 240h

# 2. Stage cockroach binary
roachprod stage $CLUSTER cockroach
# OR: roachprod put $CLUSTER ./cockroach-$SHA ./cockroach

# 3. Stage workload binary on workload node
roachprod stage $CLUSTER:$WORKLOAD_NODE workload

# 4. Start CRDB
roachprod start $CLUSTER:1-$NODES --secure
sleep 15

# 5. Create tpcc database
roachprod sql $CLUSTER:1 --secure -- -e "CREATE DATABASE IF NOT EXISTS tpcc;"

# 6. Get connection URL
PGURL=$(roachprod pgurl $CLUSTER:1 --secure | sed "s/^'//; s/'$//")

# 7. Initialize TPC-C with workload tool (CRDB kit)
roachprod ssh $CLUSTER:$WORKLOAD_NODE -- \
"./workload init tpcc --warehouses=$WAREHOUSES '$PGURL'"

# 8. Run TPC-C workload
roachprod ssh $CLUSTER:$WORKLOAD_NODE -- \
"./workload run tpcc \
--warehouses=$WAREHOUSES \
--workers=$WORKERS \
--wait=false \
--duration=$DURATION \
'$PGURL'"

# 9. Check for large ranges
roachprod ssh $CLUSTER:1-$NODES -- \
"grep -E 'initiating a split.*above threshold' logs/cockroach.log | tail -20"

# 10. Destroy
roachprod destroy $CLUSTER
```

Jira issue: CRDB-58876

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.