cockroachdb / cockroachdb/cockroach

jobs: crdb_internal.jobs and crdb_internal.system_jobs perform poorly

Open
#122,687 1 comment 0 reactions 0 assignees View on GitHub
A-jobs C-bug P-3 T-jobs
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

As part of a [larger project](https://github.com/cockroachdb/cockroach/pull/82638) to reduce job system contention in the long term. The payload and progress of all jobs were moved from `system.jobs` to `system.job_info`.

While the eventual goal is that control interfaces would not require touching `system.job_info`, that is not true for the interim state we are currently in.

Most notably, `crdb_internal.system_jobs` and `crdb_internal.jobs` (which power commands such as `SHOW JOBS` and the Jobs page on the DB Console) require reading from `system.job_info`. And, in the worst case, requires 2 full scans of system.jobs_info which can be several GB in size in large clusters.

The long-term fix for this is two-fold:

1. Move data required to show basic job status out of the job's payload and progress and into a different table.
2. Re-write crdb_internal.system_jobs and crdb_internal.jobs as virtual views rather than virtual tables. A virtual view allows the optimizer to plan much more efficient queries and should be able to eliminate the full scans in many more cases.

Ongoing work towards this plan

- [x] https://github.com/cockroachdb/cockroach/pull/123848 Avoid CTE which pessimises query plan (folded into below PR as well)
- [ ] https://github.com/cockroachdb/cockroach/pull/119081

Related work-arounds and issues:

- [x] https://github.com/cockroachdb/cockroach/pull/118638 (backport still required)
- [x] https://github.com/cockroachdb/cockroach/pull/108093
- [x] https://github.com/cockroachdb/cockroach/pull/108080
- [ ] https://github.com/cockroachdb/cockroach/pull/119574
- [x] https://github.com/cockroachdb/cockroach/pull/119355

Jira issue: CRDB-38028

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.