cockroachdb / cockroachdb/cockroach

rangefeed: registrations metric is not always drained on processor termination

Open
#106,126 3 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed A-kv-replication C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Rangefeeds maintain a metric `kv.rangefeed.registrations` which shows how many range feeds are active.
This metric is a gauge increased when registration is successfully created and must be decreased when registration is removed.

In practice, registrations could be terminated by client (when stream is closed from kv client side) or by server (when replica is removed due to rebalancing or split/merge operations).
In first case registration will terminate its output loop, which will trigger unregistration request to processor and it will perform a cleanup as a part of its work loop. Processor will then wind down itself if that was the last registration.
However, if replica decides to terminate rangefeeds, it will send stop request to processor, which will in turn terminate its registrations. Registrations will update their state and close their output loop, which would trigger unregistration request to processor, but it won't be processed because processor's work loop is already terminated.

**Environment:**
- CockroachDB version 23.1 but likely earlier versions as well.

**Additional context**
Metrics issue makes investigations problematic.

Jira issue: CRDB-29415

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.