cockroachdb / cockroachdb/cockroach

jobs: misleading error can return to user when job enters paused state

Open
#129,588 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-release-22.2 C-bug P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The `resumeContext` passed to `stepThroughStateMachine` initially here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/adopt.go#L451

may get cancelled _right after_ the state machine moves a job from running to pause requested here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/registry.go#L1658

As soon as the job's status has moved to `PauseRequested`, even before the `PauseRequested()` func has returned to the client, the pause/cancel loop will cancel the resumer ctx here:
https://github.com/msbutler/cockroach/blob/butler-pcr-mixed-version/pkg/jobs/adopt.go#L542

We observed this is some unit test flakes: e.g. https://github.com/cockroachdb/cockroach/issues/128745#issuecomment-2296786847

The `PauseRequested()` will error after txn commit as it returns back to the client, explaining the error we see in test flakes
```
pq: aborted in DistSender: result is ambiguous: context canceled
```

A long term solution will may involve passing a different context to state machine operations.

Jira issue: CRDB-41604

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.