cockroachdb / cockroachdb/cockroach
cdc: max_behind_nanos does not update when jobs are stuck during initial scans
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
See https://github.com/cockroachlabs/support/issues/2053#issuecomment-1427839384.
We only update this metric when we start emitting resolved events for changefeeds. Say one changefeed gets stuck during the initial scan (we emit no resolved events until the initial scan finishes), we won't update the `max_behind_nanos` metric and customers won't know the changefeed is stuck.
We could initialize changefeed jobs running with an initial scan with some seed highwater / resolved event.
Also related: https://github.com/cockroachdb/cockroach/issues/93919
Jira issue: CRDB-24482
Epic CRDB-8669
Contributor guide
Research direction
Start by reading the linked support comment and related issue 93919, then trace how changefeed jobs update max_behind_nanos when resolved events are emitted. Done means a changefeed stuck during its initial scan updates this metric so customers can detect the stalled progress, with tests covering that scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100