cockroachdb / cockroachdb/cockroach

changefeedccl: potential duplicate resolved messages when resolved=0s

Open
#153,462 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When the resolved option is set to 0 (which is the default value), duplicate resolved messages can be sent if the highwater is not advancing.

This is because the logic for checking if we can emit another resolved timestamp will check 1) that we've checkpointed progress and 2) that the resolved timestamp has advanced by at least the option value (0 in this case). The first case will true if we've created a span-level checkpoint even if the highwater has not advanced.

Code pointers:
* Checking if progress has been checkpointed
https://github.com/cockroachdb/cockroach/blob/a64757f26ef6c8dc9bb84f1c9cb2f9da49ce962f/pkg/ccl/changefeedccl/changefeed_processors.go#L1737
* Checking if resolved timestamp has advanced by option value
https://github.com/cockroachdb/cockroach/blob/a64757f26ef6c8dc9bb84f1c9cb2f9da49ce962f/pkg/ccl/changefeedccl/changefeed_processors.go#L2224

Repro steps:
```
SET CLUSTER SETTING changefeed.span_checkpoint.interval = '1ms';
SET CLUSTER SETTING changefeed.span_checkpoint.lag_threshold = '1ms';
CREATE TABLE large (id PRIMARY KEY) AS SELECT generate_series(1, 100000);
ALTER TABLE large SPLIT AT SELECT id FROM large ORDER BY random() LIMIT 5;
CREATE CHANGEFEED FOR large WITH initial_scan ='no', resolved;

{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692330279202000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692330279202000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692330279202000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692330279202000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692332000000000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692332000000000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692332000000000.0000000000\"}"}
{"key":"NULL","table":"NULL","value":"{\"resolved\":\"1757692332000000000.0000000000\"}"}
```

Jira issue: CRDB-54434

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.