cockroachdb / cockroachdb/cockroach
jobs: info table lifecycle
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently the info table rows for a job are deleted when the job is deleted -- typically that is when it is in a terminal state and 14 days have elapsed from its finish time.
However this means that if any code produces info rows at a steady rate while a job is running and does not delete them, if that job that runs indefinitely, like a changefeed or span reconciliation, then those info rows will continue to be accumulated indefinitely.
We potentially should identify an info key prefix to denote ephemeral data, for example for all keys created automatically by job-system-internal observability features, and teach the job system to delete ephemeral data that is older than some limit (probably equal to the job retention number). We would potentially want to only do this for those jobs in active states (running or reverting) that have this potential to be continually accumulating data, and were created more than than retention period ago, to avoid doing a full-table scan on the info table.
An alternative would be to write a ready-to-use cleanup loop that can be started for any given job id to find and expire info rows, which the resumer for any perpetual job would then be responsible for calling. If all perpetual info row appends are voluntary / opt-in by a job of a given type, making the cleanup similarly opt-in seems reasonable, but if we ever start accumulating infos automatically for running jobs then we likely want the registry to be responsible for doing it for the job, either on a per-job basis (each registry is responsible for running the loop for all its active jobs) or globally (join active jobs to infos).
Jira issue: CRDB-27567
Contributor guide
Assessment
This issue has not been assessed yet.