cockroachdb / cockroachdb/cockroach

jobs: failed resumer is reused for OnFailOrCancel while its goroutines are still live

Open
#174,959 1 comment 0 reactions 0 assignees View on GitHub
A-jobs C-bug O-agent T-jobs
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**

When a job resumer fails in a way that transitions the job to reverting, `jobs.Registry.stepThroughStateMachine` recurses with the *same* `resumer` value, so `OnFailOrCancel` runs on an object that the failed `Resume` may have left half-mutated. Separately, `aj.cancel()` fires only once `runJob` returns, so goroutines the failed `Resume` spawned are still alive and still writing through the resumer's state while `OnFailOrCancel` runs on it. Part of #174944.

This matters most if a `Resume` failure is ever handled without tearing the process down, since in that case the half-mutated resumer is reused rather than discarded.

**Findings:**

- `jobs.Registry.stepThroughStateMachine`: `resumer` — reused for `OnFailOrCancel` after a failed `Resume` rather than reconstructed.
- `jobs.Registry` / `adopt`: the resumer's goroutines are not cancelled before `OnFailOrCancel` begins.

**Next Steps:**

- [ ] Call `r.createResumer(job)` before entering the reverting path rather than reusing the existing resumer
- [ ] Cancel the job's goroutines before invoking `OnFailOrCancel`, or document why the overlap is safe

Epic: none

Jira issue: CRDB-68130

Contributor guide

Open the contributing guide

Research direction

Start in jobs.Registry.stepThroughStateMachine and trace how the resumer reaches the reverting path, then inspect jobs.Registry/adopt, runJob, and aj.cancel for goroutine lifetime. Check how r.createResumer(job) and OnFailOrCancel interact. Done means the reverting path does not reuse a failed resumer and its goroutines cannot overlap unsafely with OnFailOrCancel, or the safety is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.