cockroachdb / cockroachdb/cockroach

SHOW BACKUPS can cause node to OOM

Open
#102,892 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**
The query used by the backups page in the CC console can cause an OOM on the node planning the `SHOW BACKUPS` query.

Heap profile on the OOMing node right before the OOM shows almost all of the memory is being used on reading backup manifests.
![image](https://user-images.githubusercontent.com/17620970/236885740-de930367-7b65-408c-a9c1-1e71b249d6b6.png)

The logic around the backup manifests should be [memory monitored](https://github.com/cockroachdb/cockroach/blob/v22.2.9/pkg/ccl/backupccl/backupinfo/manifest_handling.go#L268), thus it is unclear why the query doesn't simply fail due to a memory monitor error instead of causing an OOM.

[Internal support issue](https://github.com/cockroachlabs/support/issues/2288)

**Expected behavior**
`SHOW BACKUPS` query should fail when there is a lack of memory to read the backup manifests instead of OOMing.

**Environment:**
- CockroachDB version [22.2.7]

Jira issue: CRDB-27712

Contributor guide

Open the contributing guide

Research direction

Start by reading pkg/ccl/backupccl/backupinfo/manifest_handling.go around line 268, then trace how SHOW BACKUPS reads backup manifests and applies memory monitoring. Confirm the behavior under low memory and determine why the query OOMs instead of failing with a memory monitor error; done means the expected failure occurs without an OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.