cockroachdb / cockroachdb/cockroach

CDC: support resolved messages with db-level feeds for pubsub and kafka

Open
#158,408 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

*Description*

All db-level changefeeds set split column families which is not compatible with resolved for pubsub and kafka.

See this code in changefeed_stmt:

if opts.IsSet(changefeedbase.OptResolvedTimestamps) &&
opts.IsSet(changefeedbase.OptSplitColumnFamilies) {
return errors.Newf("Resolved timestamps are not currently supported with %s for this sink"+
" as the set of topics to fan them out to may change. Instead, use TABLE tablename FAMILY familyname"+
" to specify individual families to watch.", changefeedbase.OptSplitColumnFamilies)
}

To fix, we'd need to make changes to the topic namer to dynamically generate a list of topics.

Then we'd be able to add coverage more coverage with DB-level feeds metamorphic testing.
We may also be able to add this ability for table level feeds.

Jira issue: CRDB-57276

Epic CRDB-1421

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.