cockroachdb / cockroachdb/cockroach
backupccl: restore fails when an addSSTable triggers a failed backpressure split
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A backup with large keys can fail to restore due to a failed addSSTable request:
```
I230425 14:47:49.672996 182784 jobs/registry.go:1215 ⋮ [n1] 1522 RESTORE job 859713252283613185: stepping through state reverting with error: importing 8 ranges: addsstable [/Table/109/1/‹3›/‹0›,/Table/109/1/‹3›/ ‹0›/‹NULL›): split failed while applying backpressure to AddSSTable [/Table/109/1/‹3›/‹0›,/Table/109/1/‹3›/‹0›/‹NULL›) on range r83:‹/Table/10{6/1/9-9/1/4}› [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=13]: ‹could not find valid split key›
```
It seems that this specific AddSSTable request had a single key, as the start and end keys are identical. Further, there was an [attempt ](https://github.com/cockroachdb/cockroach/blob/7503b1a109e8951a7172657adea48dd5ee855cf9/pkg/kv/kvserver/replica_backpressure.go#L197)by the kv server to split the requests' receiving range via `maybeBackpressureBatch()`. Note that in this repro, each key in the backup was 2 MB, and generated in this WIP [roachtest](https://github.com/cockroachdb/cockroach/pull/102040) (note, use only this [commit ](https://github.com/cockroachdb/cockroach/pull/102040/commits/d61999325a8c08a84b4b96be23e069f343d64aaf)in the PR).
We should figure out why the split failed, and consequently, how to handle this failure. Restore shouldn't fail just because the keys are kinda big. The [public docs](https://www.cockroachlabs.com/docs/stable/common-errors.html#split-failed-while-applying-backpressure-are-rows-updated-in-a-tight-loop) state this error could surface if there are many updates to a given row, but this doesn't apply here because restore only writes a single mvcc version of each key.
Debug zip of failed roachtest [here](https://drive.google.com/drive/u/0/folders/1VHA5IIRoE49w97flijVacBpIeqCT9SiZ).
I have observed this on 22.2 and 23.1. I have yet to test master or 22.1.
Jira issue: CRDB-27380
Contributor guide
Assessment
This issue has not been assessed yet.