cockroachdb / cockroachdb/cockroach
jobs.running_non_idle can go negative
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The metric `jobs.running_non_idle` was observed to report negative values in production. For Serverless, we noticed this occurring when sql servers were gracefully draining.
**Reproduction Instructions**
- start a sql server and wait for all of the jobs to enter an idle state.
- send a sigterm to the sql server to start the graceful drain process.
- curl metrics in a loop until you see negative value for `jobs.running_non_idle`
**Investigation**
This appears to be a side effect of calling [cancelAllAdoptedJobs](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1858) while there are idle jobs. `cancelAllAdoptedJobs` cancels the context for every job and removes all jobs from the `adoptedJobs` map. [MarkIdle](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1830) does not update the idle count metric if a job is missing from the `adoptedJobs` map. Cancelling the jobs cause [Resume to return which decrements the running idle job count](https://github.com/cockroachdb/cockroach/blob/6f420390c1f9672ecad250dde8bf53da7f9da8e2/pkg/jobs/registry.go#L1646). Once Resume returns, `r.MarkIdle(jobs, false)` is called which would normally fix the metric value, but since the `adoptedJobs` map is empty, the `MarkIdle` call is a no-op.
`cancelAllAdoptedJobs` is also called when during the graceful drain process and if the sql liveness session expires. For non-serverless deployments an expired sql session is a recoverable state, so the the metric could end up negative on a long running process.
Contributor guide
Assessment
This issue has not been assessed yet.