cockroachdb / cockroachdb/cockroach

kvserver/rangefeed: RangeFeedProcessors gauge can go negative when Start fails after scheduler registration

Open
#172,420 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-master C-bug O-agent P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

`ScheduledProcessor.Start` increments `RangeFeedProcessors` only after all its fallible steps ([scheduled_processor.go#L152](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduled_processor.go#L152)), but the error paths in between (rtsIter construction failure, `RunAsyncTask` failure) call `p.scheduler.StopProcessor()` after the scheduler registration has already succeeded. The eventual `processStop` decrements the gauge unconditionally ([scheduled_processor.go#L254](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduled_processor.go#L254)), so each such failure drives the gauge one below reality, permanently.

Metric-only impact.

**Suggested fix**

Move the `Inc` before the fallible steps, or make `processStop` decrement only when `Start` completed.

**Additional context**

Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.

Jira issue: CRDB-65656

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvserver/rangefeed/scheduled_processor.go, reading ScheduledProcessor.Start around line 152 and processStop around line 254. Trace the rtsIter construction and RunAsyncTask failure paths after scheduler registration, then verify that each failed start leaves RangeFeedProcessors accurate and unable to go negative.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.