cockroachdb / cockroachdb/cockroach
cloud: external storage read path (ReadFile/ReadAt) has no timeout and can hang indefinitely
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The external storage `ReadFile` path — which backs the `ReadAt`/sstable read path used by backup compaction, restore, etc. — does not apply any application-level timeout to the underlying object read. If a read stalls (e.g. the request is sent but response headers never arrive), the call blocks indefinitely. Because no error is ever returned, the resume/retry logic (`ResumingReader`, `cloudstorage.s3.max_retries`) never engages, and the reading processor — and thus the job — sits in `running` with `fraction_completed` frozen and `no_progress_for` growing without bound.
This is asymmetric with `Size` and `Delete`, which already wrap their calls in `timeutil.RunWithTimeout(ctx, ..., cloud.Timeout.Get(...))` (the `cloudstorage.timeout` setting). `ReadFile` does not, so `cloudstorage.timeout` does not cover streaming reads.
**The gap is present in all three supported providers** (read open is unbounded; `Size`/`Delete` are bounded):
- **S3** — [`openStreamAt` → `GetObject` (s3_storage.go:942)](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/amazon/s3_storage.go#L942); cf. [`Size` L1159](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/amazon/s3_storage.go#L1159), [`Delete` L1138](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/amazon/s3_storage.go#L1138).
- **GCS** — [`ReadFile` → `NewRangeReader` (gcs_storage.go:447)](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/gcp/gcs_storage.go#L447); cf. [`Size` L520](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/gcp/gcs_storage.go#L520), [`Delete` L508](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/gcp/gcs_storage.go#L508).
- **Azure** — [`ReadFile` → `DownloadStream` (azure_storage.go:432)](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/azure/azure_storage.go#L432); cf. [`Size` L518](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/azure/azure_storage.go#L518), [`Delete` L505](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/azure/azure_storage.go#L505).
The read is reached provider-agnostically via [`sstReader.ReadAt` → `openAt` → `Store.ReadFile` (external_sst_reader.go:154)](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/storage/external_sst_reader.go#L154), and the shared [`ResumingReader`](https://github.com/cockroachdb/cockroach/blob/f729dc8d6e1160440583f7bac9853d392f3376ab/pkg/cloud/cloud_io.go) only retries on a *returned* error, so a hung open/read defeats it identically for every provider.
**Suggested fix / design constraint**
Apply the bound consistently across the three `ReadFile` implementations (or in shared `ResumingReader`), not just one provider. The bound must target **time-to-first-byte / per-`Read` stall**, not total transfer time — a naive `RunWithTimeout` around the whole streaming read would break large legitimate reads.
Jira issue: CRDB-66127
Contributor guide
Assessment
This issue has not been assessed yet.