cockroachdb / cockroachdb/cockroach
rangefeed: registrations metric is not always drained on processor termination
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Rangefeeds maintain a metric `kv.rangefeed.registrations` which shows how many range feeds are active.
This metric is a gauge increased when registration is successfully created and must be decreased when registration is removed.
In practice, registrations could be terminated by client (when stream is closed from kv client side) or by server (when replica is removed due to rebalancing or split/merge operations).
In first case registration will terminate its output loop, which will trigger unregistration request to processor and it will perform a cleanup as a part of its work loop. Processor will then wind down itself if that was the last registration.
However, if replica decides to terminate rangefeeds, it will send stop request to processor, which will in turn terminate its registrations. Registrations will update their state and close their output loop, which would trigger unregistration request to processor, but it won't be processed because processor's work loop is already terminated.
**Environment:**
- CockroachDB version 23.1 but likely earlier versions as well.
**Additional context**
Metrics issue makes investigations problematic.
Jira issue: CRDB-29415
Contributor guide
Assessment
This issue has not been assessed yet.