cockroachdb / cockroachdb/cockroach

jobs,*: address sources of bloat in jobs tables

Open
#155,448 1 comment 0 reactions 0 assignees View on GitHub
C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We've observed some clusters have a larger number of retained jobs than usual, or in some cases a larger number of retained job_info rows in particular.

- [ ] *AUTO CREATE STATS* jobs are created and run _frequently_ and are by far the largest contributor to the number of retained jobs in most clusters (60%+)
- [ ] *ROW LEVEL TTL* jobs are often created automatically at high frequency -- albeit less frequently than auto stats -- in many cases multiplied by a number of tables configured to use TTLs.
- [ ] *BACKUP* jobs similarly are created by a schedule, though the most common configuration only has a single full and inc schedule per cluster rather than multiplying the schedule frequency by a number of tables.
- [ ] *BACKUP* jobs have additionally been observed to be bloating the info rows more _per job_ due to ~profiler/ rows with per-node progress; these do not need to be retained beyond job completion (... or at all?)
- [ ] *Orphaned job infos* Older versions didn't transactionally delete infos when they deleted a job, so they could be orphaned; we thought this was minor enough to ignore, but it could be adding up in clusters that restarted nodes often or had contention jobs.
- [ ] CDC jobs still running with old diagram infos; we can _probably_ just ignore these since those old infos will get cleaned up when the jobs eventually terminate, but maybe it is worth adding the old name pattern to the pruning to get them sooner.

Jira issue: CRDB-55440

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.