cockroachdb / cockroachdb/cockroach
kvserver/rangefeed: scheduler leaks a status map entry per processor lifetime
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
When a scheduler worker processes a `Stopped` event, it writes the status back unconditionally after the callback returns ([scheduler.go#L509-L511](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L509-L511)), unlike the non-Stopped path which first checks that the id is still registered ([scheduler.go#L516](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L516)):
1. `stopInternal` enqueues `Stopped` for processor id N.
2. A shard worker pops N, sets `status[N] = Queued|Stopped`, and invokes the callback.
3. The callback runs `processStop` → `cleanup` → `ClientScheduler.Unregister`, which deletes N from both `ss.procs` and `ss.status` ([scheduler.go#L365-L370](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvserver/rangefeed/scheduler.go#L365-L370)).
4. The callback returns; the worker executes `ss.status[N] = Stopped`, re-creating the deleted entry.
5. Nothing ever removes it: `enqueueLocked` bails at the `ss.procs` lookup, the `newStatus == Queued` delete requires N to be popped again (impossible), and `shard.stop()` ranges over `ss.procs` only. Processor ids are monotonically allocated and never reused.
Every processor start/stop cycle (changefeed reconnects, splits/merges, last-registration-gone shutdowns) permanently leaks one `map[int64]processorEventType` entry in the shard for the life of the store. Slow, but unbounded.
**Suggested fix**
Mirror the non-Stopped path: only write `Stopped` back if the id is still present in `ss.procs`.
**Additional context**
Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.
Jira issue: CRDB-65654
Contributor guide
Research direction
Read pkg/kv/kvserver/rangefeed/scheduler.go around lines 365-370 and 509-516, then trace stopInternal, processStop, cleanup, and ClientScheduler.Unregister. Verify the stopped-event path only restores status when the processor id remains registered in ss.procs, and confirm that stopped processors no longer leave entries in ss.status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100