cockroachdb / cockroachdb/cockroach

restore: RESTORE from azure hung at 96% for hours

Open
#110,031 2 comments 1 reaction 0 assignees View on GitHub
A-disaster-recovery C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A relatively small restore -- only 29 import spans, writing somewhere below 11gb replicated, using 5 nodes -- was observed to have run for around 5-7 minutes and reached 96% complete before then not updating again for between 20 minutes and over an hour.

Two examples of such a restore were found on the same cluster. Analysis of the stacks on the nodes while the restore was "hung" (not updating its progress) showed that in both cases, all split-and-scatter processors and but one restore data processor had exited. The one remaining restore data processor in both cases was reading from azure, in one case in `openSSTs` and the other in `processRestoreSpanEntry`, eventually ending up in `IO wait` or `select` in an http read coming from azblob.

It is unclear what rate these azblob reads are happening at, if they are hung or are being retried or something else is going on.
[goroutine.11081.txt](https://github.com/cockroachdb/cockroach/files/12525040/goroutine.11081.txt)
[goroutine.113855.txt](https://github.com/cockroachdb/cockroach/files/12525048/goroutine.113855.txt)

@dt and @benbardin have additional analysis in on internal documents related to support incident.

Jira issue: CRDB-31232

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.