cockroachdb / cockroachdb/cockroach
jobs: misleading error can return to user when job enters paused state
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The `resumeContext` passed to `stepThroughStateMachine` initially here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/adopt.go#L451
may get cancelled _right after_ the state machine moves a job from running to pause requested here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/registry.go#L1658
As soon as the job's status has moved to `PauseRequested`, even before the `PauseRequested()` func has returned to the client, the pause/cancel loop will cancel the resumer ctx here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/adopt.go#L542
We observed this is some unit test flakes: e.g. https://github.com/cockroachdb/cockroach/issues/128745#issuecomment-2296786847
The `PauseRequested()` will error after txn commit as it returns back to the client, explaining the error we see in test flakes
```
pq: aborted in DistSender: result is ambiguous: context canceled
```
A long term solution will may involve passing a different context to state machine operations.
Jira issue: CRDB-41604
Contributor guide
Assessment
This issue has not been assessed yet.