cockroachdb / cockroachdb/cockroach

kvserver/rangefeed: scheduler leaks a status map entry per processor lifetime

Open Beginner friendly
#172,418 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-master C-bug O-agent P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When a scheduler worker processes a `Stopped` event, it writes the status back unconditionally after the callback returns ([scheduler.go#L509-L511](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L509-L511)), unlike the non-Stopped path which first checks that the id is still registered ([scheduler.go#L516](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L516)):

1. `stopInternal` enqueues `Stopped` for processor id N.
2. A shard worker pops N, sets `status[N] = Queued|Stopped`, and invokes the callback.
3. The callback runs `processStop` → `cleanup` → `ClientScheduler.Unregister`, which deletes N from both `ss.procs` and `ss.status` ([scheduler.go#L365-L370](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L365-L370)).
4. The callback returns; the worker executes `ss.status[N] = Stopped`, re-creating the deleted entry.
5. Nothing ever removes it: `enqueueLocked` bails at the `ss.procs` lookup, the `newStatus == Queued` delete requires N to be popped again (impossible), and `shard.stop()` ranges over `ss.procs` only. Processor ids are monotonically allocated and never reused.

Every processor start/stop cycle (changefeed reconnects, splits/merges, last-registration-gone shutdowns) permanently leaks one `map[int64]processorEventType` entry in the shard for the life of the store. Slow, but unbounded.

**Suggested fix**

Mirror the non-Stopped path: only write `Stopped` back if the id is still present in `ss.procs`.

**Additional context**

Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.

Jira issue: CRDB-65654

Contributor guide

Open the contributing guide

Research direction

Read pkg/kv/kvserver/rangefeed/scheduler.go around lines 365-370 and 509-516, then trace stopInternal, processStop, cleanup, and ClientScheduler.Unregister. Verify the stopped-event path only restores status when the processor id remains registered in ss.procs, and confirm that stopped processors no longer leave entries in ss.status.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.