cockroachdb / cockroachdb/cockroach

jobs/backups: database level backups of non-trivial schema's can lead to regular memory spikes and OOMs

Open
#111,879 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug O-support P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Inspired from https://github.com/cockroachlabs/support/issues/2625

A database schema with a couple thousand descriptors can lead to backup job payload objects in the 10's of MiB territory. Due to the jobs retention policy of 14 days an aggressive backup schedule can lead to job/jobs_info table containing GiB's of these payload objects and the jobs GC job has to select and unpack these payload objects to determine which jobs to prune (happens once an hour). The memory impact of this GC activity on a jobs table with ~100 backups can lead to GiB's of memory allocated. These allocations happen at multiple levels (storage/kv/SQL//internal protobuf decoding) at least half dozen times when the payload objects are selected. Therefore the easiest fix is probabaly pull the necessary metadata out of the payload so jobs can be queried for GC w/o being selected.

Another potentially more general purpose fix would be to look at how the jobs GC query is retaining memory in the presence of large rows and see if we can do something to allow memory to be GC'd quicker so the memory spikes aren't as big.

Jira issue: CRDB-32107

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.