cockroachdb / cockroachdb/cockroach
jobs: failed resumer is reused for OnFailOrCancel while its goroutines are still live
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary:**
When a job resumer fails in a way that transitions the job to reverting, `jobs.Registry.stepThroughStateMachine` recurses with the *same* `resumer` value, so `OnFailOrCancel` runs on an object that the failed `Resume` may have left half-mutated. Separately, `aj.cancel()` fires only once `runJob` returns, so goroutines the failed `Resume` spawned are still alive and still writing through the resumer's state while `OnFailOrCancel` runs on it. Part of #174944.
This matters most if a `Resume` failure is ever handled without tearing the process down, since in that case the half-mutated resumer is reused rather than discarded.
**Findings:**
- `jobs.Registry.stepThroughStateMachine`: `resumer` — reused for `OnFailOrCancel` after a failed `Resume` rather than reconstructed.
- `jobs.Registry` / `adopt`: the resumer's goroutines are not cancelled before `OnFailOrCancel` begins.
**Next Steps:**
- [ ] Call `r.createResumer(job)` before entering the reverting path rather than reusing the existing resumer
- [ ] Cancel the job's goroutines before invoking `OnFailOrCancel`, or document why the overlap is safe
Epic: none
Jira issue: CRDB-68130
Contributor guide
Research direction
Start in jobs.Registry.stepThroughStateMachine and trace how the resumer reaches the reverting path, then inspect jobs.Registry/adopt, runJob, and aj.cancel for goroutine lifetime. Check how r.createResumer(job) and OnFailOrCancel interact. Done means the reverting path does not reuse a failed resumer and its goroutines cannot overlap unsafely with OnFailOrCancel, or the safety is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100