cockroachdb / cockroachdb/cockroach
jobs/backups: database level backups of non-trivial schema's can lead to regular memory spikes and OOMs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Inspired from https://github.com/cockroachlabs/support/issues/2625
A database schema with a couple thousand descriptors can lead to backup job payload objects in the 10's of MiB territory. Due to the jobs retention policy of 14 days an aggressive backup schedule can lead to job/jobs_info table containing GiB's of these payload objects and the jobs GC job has to select and unpack these payload objects to determine which jobs to prune (happens once an hour). The memory impact of this GC activity on a jobs table with ~100 backups can lead to GiB's of memory allocated. These allocations happen at multiple levels (storage/kv/SQL//internal protobuf decoding) at least half dozen times when the payload objects are selected. Therefore the easiest fix is probabaly pull the necessary metadata out of the payload so jobs can be queried for GC w/o being selected.
Another potentially more general purpose fix would be to look at how the jobs GC query is retaining memory in the presence of large rows and see if we can do something to allow memory to be GC'd quicker so the memory spikes aren't as big.
Jira issue: CRDB-32107
Contributor guide
Assessment
This issue has not been assessed yet.