cockroachdb / cockroachdb/cockroach
SHOW BACKUPS can cause node to OOM
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The query used by the backups page in the CC console can cause an OOM on the node planning the `SHOW BACKUPS` query.
Heap profile on the OOMing node right before the OOM shows almost all of the memory is being used on reading backup manifests.

The logic around the backup manifests should be [memory monitored](https://github.com/cockroachdb/cockroach/blob/v22.2.9/pkg/ccl/backupccl/backupinfo/manifest_handling.go#L268), thus it is unclear why the query doesn't simply fail due to a memory monitor error instead of causing an OOM.
[Internal support issue](https://github.com/cockroachlabs/support/issues/2288)
**Expected behavior**
`SHOW BACKUPS` query should fail when there is a lack of memory to read the backup manifests instead of OOMing.
**Environment:**
- CockroachDB version [22.2.7]
Jira issue: CRDB-27712
Contributor guide
Research direction
Start by reading pkg/ccl/backupccl/backupinfo/manifest_handling.go around line 268, then trace how SHOW BACKUPS reads backup manifests and applies memory monitoring. Confirm the behavior under low memory and determine why the query OOMs instead of failing with a memory monitor error; done means the expected failure occurs without an OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100