cockroachdb / cockroachdb/cockroach

kvcoord: mux rangefeed metadata can arrive after events from the partial rangefeed it announces

Open
#172,423 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-master C-bug O-agent P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

`muxRangeFeed.startSingleRangeFeed` sends the metadata event only after `stream.start()` succeeds ([dist_sender_mux_rangefeed.go#L230-L236](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go#L230-L236)). By that point, `receiveEventsFromNode` may already be delivering that stream's data and checkpoint events to the shared event channel from another goroutine, so consumers can observe events for a partial rangefeed before the metadata that announces it.

`OnMetadata`'s documented contract says the callback "occurs when a partial rangefeed begins" ([config.go#L242-L245](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/config.go#L242-L245)), which implies begins-before-events ordering that the code does not provide. For the logical-data-replication consumer, which uses metadata to install manual split points, a split point can trail data from the child span.

**Suggested fix**

Either hold event delivery for a stream until its metadata has been emitted (e.g. emit metadata as part of the registration step on the receiver path), or explicitly document that metadata ordering relative to the stream's events is unspecified and audit consumers for that assumption.

**Additional context**

Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.

Jira issue: CRDB-65659

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go, especially muxRangeFeed.startSingleRangeFeed and receiveEventsFromNode, then review the OnMetadata contract in pkg/kv/kvclient/rangefeed/config.go. Trace how metadata and data/checkpoint events reach the shared channel. Done means the ordering contract is made reliable, or the contract is explicitly documented and affected consumers are audited.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.