cockroachdb / cockroachdb/cockroach

cdc: max_behind_nanos does not update when jobs are stuck during initial scans

Open
#97,043 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

See https://github.com/cockroachlabs/support/issues/2053#issuecomment-1427839384.

We only update this metric when we start emitting resolved events for changefeeds. Say one changefeed gets stuck during the initial scan (we emit no resolved events until the initial scan finishes), we won't update the `max_behind_nanos` metric and customers won't know the changefeed is stuck.

We could initialize changefeed jobs running with an initial scan with some seed highwater / resolved event.

Also related: https://github.com/cockroachdb/cockroach/issues/93919

Jira issue: CRDB-24482

Epic CRDB-8669

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked support comment and related issue 93919, then trace how changefeed jobs update max_behind_nanos when resolved events are emitted. Done means a changefeed stuck during its initial scan updates this metric so customers can detect the stalled progress, with tests covering that scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.