cockroachdb / cockroachdb/cockroach

jobs: job fails to start despite running state

Open
#119,199 0 comments 0 reactions 0 assignees View on GitHub
A-jobs C-bug P-2 T-jobs
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In at least 2 escalations, we've seen cases where an `AUTO CREATE STATS` job has failed to start for a prolonged period of time.

In the case with more detailed debug information we know that:

- The starteMicros in the job payload was never set. This is set right before the resumer is called.
- The job was claimed by with a valid lease.
- Cancelling the job moved the job to reverting but the job never entered the canceled state.

Based on this behaviour, we suspect that the job was added to the registries adoptedJobs array but never removed after some error occurred. One possible place that happens is here:

https://github.com/cockroachdb/cockroach/blob/254dbd247fb8ed352a11439063b29f23a0767f28/pkg/jobs/jobs.go#L969-L972

If we happen to get a non-context-cancellation-related error that still resulted in the local transaction state being recorded as Committed (I don't know how likely that is).

From the looks of it, it should be safe to remove the job from the adoptedJobs map even in this case. Even if that condition is catching a programming error, the user has indicated that they aren't going to Start() the job. So we should remove it from the adopted array.

Jira issue: CRDB-36086

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.