cockroachdb / cockroachdb/cockroach

jobs.running_non_idle can go negative

Open
#107,119 1 comment 0 reactions 0 assignees View on GitHub
A-jobs C-bug T-jobs T-serverless
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The metric `jobs.running_non_idle` was observed to report negative values in production. For Serverless, we noticed this occurring when sql servers were gracefully draining.

**Reproduction Instructions**

- start a sql server and wait for all of the jobs to enter an idle state.
- send a sigterm to the sql server to start the graceful drain process.
- curl metrics in a loop until you see negative value for `jobs.running_non_idle`

**Investigation**

This appears to be a side effect of calling [cancelAllAdoptedJobs](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1858) while there are idle jobs. `cancelAllAdoptedJobs` cancels the context for every job and removes all jobs from the `adoptedJobs` map. [MarkIdle](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1830) does not update the idle count metric if a job is missing from the `adoptedJobs` map. Cancelling the jobs cause [Resume to return which decrements the running idle job count](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1646). Once Resume returns, `r.MarkIdle(jobs, false)` is called which would normally fix the metric value, but since the `adoptedJobs` map is empty, the `MarkIdle` call is a no-op.

`cancelAllAdoptedJobs` is also called when during the graceful drain process and if the sql liveness session expires. For non-serverless deployments an expired sql session is a recoverable state, so the the metric could end up negative on a long running process.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.