cockroachdb / cockroachdb/cockroach

kv: load tests fail with "could not find valid split key" error

Open
#146,795 2 comments 1 reaction 0 assignees View on GitHub
A-vector-index C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The `vecann` workload will occasionally fail with this error:
```
E250513 23:28:02.179884 1 workload/cli/run.go:587 [-] 6 +Wraps: (3) ERROR: split failed while applying backpressure to ConditionalPut [/Table/106/1/1071869074371215361/0], ConditionalPut [/Table/106/1/1071869074371248129/0], ConditionalPut [/Table/106/1/1071869074371280897/0], [txn: d771d95b] on range r267:/Table/106/{1/1071868916027359235-2/0/1/0} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=107]: could not find valid split key (SQLSTATE XXUUU)
```
Every time this has happened the splitting range is always on the boundary between the primary index and the secondary vector index (i.e. `2/0/1/0`).

REPRO:

First build artifacts and set $EMAIL:
```
./dev build --cross
./dev build workload --cross

export EMAIL=
```

Then run the following script. It will continue looping until it fails, at which point the script will end. This can take more than 30 minutes to fail.
```
#!/bin/bash

# Update the gce-zones to use whatever region you like.
roachprod create $EMAIL-crdb-3x16 -n 4 --gce-machine-type=n2-standard-16 --local-ssd=false --geo --gce-zones='us-west4-a,us-west4-b,us-west4-c' --gce-use-spot

set -e # Exit on first error

while true; do
echo "Starting test run at $(date)"

roachprod wipe "$EMAIL-crdb-3x16"
roachprod put "$EMAIL-crdb-3x16:1-3" artifacts/cockroach
roachprod start "$EMAIL-crdb-3x16:1-3"
roachprod run "$EMAIL-crdb-3x16:1" -- ./cockroach sql --certs-dir=./certs -e "SET CLUSTER SETTING feature.vector_index.enabled=on"
roachprod put "$EMAIL-crdb-3x16:4" artifacts/workload
roachprod ssh "$EMAIL-crdb-3x16:4" -- ./workload init vecann --dataset=laion-1m-test-ip --cache-folder=/mnt/data1/.cache/workload-datasets {pgurl:1-3}
roachprod ssh "$EMAIL-crdb-3x16:4" -- ./workload run vecann --batch-size=3 --dataset=laion-1m-test-ip --search-percent=0 --num-groups=10000000 --concurrency=80 --ramp=2m --duration=5m --cache-folder=/mnt/data1/.cache/workload-datasets {pgurl:1-3}

echo "Test run completed successfully at $(date)"
echo "Restarting..."
done
```

Clean up the cluster when it's no longer needed.
```
roachprod destroy $EMAIL-crdb-3x16
```

Jira issue: CRDB-50710

Contributor guide

Open the contributing guide

Research direction

Start with the vecann workload reproduction and the split error reported at workload/cli/run.go:587; investigate the primary-index/secondary-vector-index boundary described in the report. Done means the supplied loop no longer reaches “could not find valid split key”; build the artifacts first and clean up the cluster afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.