cockroachdb / cockroachdb/cockroach
kv: load tests fail with "could not find valid split key" error
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The `vecann` workload will occasionally fail with this error:
```
E250513 23:28:02.179884 1 workload/cli/run.go:587 [-] 6 +Wraps: (3) ERROR: split failed while applying backpressure to ConditionalPut [/Table/106/1/1071869074371215361/0], ConditionalPut [/Table/106/1/1071869074371248129/0], ConditionalPut [/Table/106/1/1071869074371280897/0], [txn: d771d95b] on range r267:/Table/106/{1/1071868916027359235-2/0/1/0} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=107]: could not find valid split key (SQLSTATE XXUUU)
```
Every time this has happened the splitting range is always on the boundary between the primary index and the secondary vector index (i.e. `2/0/1/0`).
REPRO:
First build artifacts and set $EMAIL:
```
./dev build --cross
./dev build workload --cross
export EMAIL=
```
Then run the following script. It will continue looping until it fails, at which point the script will end. This can take more than 30 minutes to fail.
```
#!/bin/bash
# Update the gce-zones to use whatever region you like.
roachprod create $EMAIL-crdb-3x16 -n 4 --gce-machine-type=n2-standard-16 --local-ssd=false --geo --gce-zones='us-west4-a,us-west4-b,us-west4-c' --gce-use-spot
set -e # Exit on first error
while true; do
echo "Starting test run at $(date)"
roachprod wipe "$EMAIL-crdb-3x16"
roachprod put "$EMAIL-crdb-3x16:1-3" artifacts/cockroach
roachprod start "$EMAIL-crdb-3x16:1-3"
roachprod run "$EMAIL-crdb-3x16:1" -- ./cockroach sql --certs-dir=./certs -e "SET CLUSTER SETTING feature.vector_index.enabled=on"
roachprod put "$EMAIL-crdb-3x16:4" artifacts/workload
roachprod ssh "$EMAIL-crdb-3x16:4" -- ./workload init vecann --dataset=laion-1m-test-ip --cache-folder=/mnt/data1/.cache/workload-datasets {pgurl:1-3}
roachprod ssh "$EMAIL-crdb-3x16:4" -- ./workload run vecann --batch-size=3 --dataset=laion-1m-test-ip --search-percent=0 --num-groups=10000000 --concurrency=80 --ramp=2m --duration=5m --cache-folder=/mnt/data1/.cache/workload-datasets {pgurl:1-3}
echo "Test run completed successfully at $(date)"
echo "Restarting..."
done
```
Clean up the cluster when it's no longer needed.
```
roachprod destroy $EMAIL-crdb-3x16
```
Jira issue: CRDB-50710
Contributor guide
Research direction
Start with the vecann workload reproduction and the split error reported at workload/cli/run.go:587; investigate the primary-index/secondary-vector-index boundary described in the report. Done means the supplied loop no longer reaches “could not find valid split key”; build the artifacts first and clean up the cluster afterward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, go
- Domain
- databases, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100