cockroachdb / cockroachdb/cockroach
restore: RESTORE from azure hung at 96% for hours
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A relatively small restore -- only 29 import spans, writing somewhere below 11gb replicated, using 5 nodes -- was observed to have run for around 5-7 minutes and reached 96% complete before then not updating again for between 20 minutes and over an hour.
Two examples of such a restore were found on the same cluster. Analysis of the stacks on the nodes while the restore was "hung" (not updating its progress) showed that in both cases, all split-and-scatter processors and but one restore data processor had exited. The one remaining restore data processor in both cases was reading from azure, in one case in `openSSTs` and the other in `processRestoreSpanEntry`, eventually ending up in `IO wait` or `select` in an http read coming from azblob.
It is unclear what rate these azblob reads are happening at, if they are hung or are being retried or something else is going on.
[goroutine.11081.txt](https://github.com/cockroachdb/cockroach/files/12525040/goroutine.11081.txt)
[goroutine.113855.txt](https://github.com/cockroachdb/cockroach/files/12525048/goroutine.113855.txt)
@dt and @benbardin have additional analysis in on internal documents related to support incident.
Jira issue: CRDB-31232
Contributor guide
Assessment
This issue has not been assessed yet.