cockroachdb / cockroachdb/cockroach

cdc: filter out updates from schema changes by temporarily switching to diff feeds

Open
#106,534 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We're currently unable to fully honor the `WITH schema_change_policy='nobackfill'` option, because some schema changes cause a rangefeed to re-emit the entire table. These are becoming fewer and fewer as more schema changes become primary index swaps, but there's a long tail. We can detect schema change timestamps thanks to the schema feed, but we can't safely just ignore events with that timestamp because it's possible to have actual data changes at the same timestamp as a schema change (e.g. if a transaction includes both DDL and DML).

However! A kvfeed with the WithDiff flag set to true does have enough information to filter out no-op updates--if the the before and after values are identical, it could suppress the event. So I propose we add a check in all changefeeds where `schema_change_policy='nobackfill'`: if an event has the same timestamp as a schema change, we only process it if the before and after values differ. If the changefeed is running without diff, it will temporarily switch to a diff kvfeed just for the timestamp of the schema change. This switchover may be a bit expensive in CPU, but should be much less expensive than primary index swaps are for changefeeds.

Jira issue: CRDB-29606

Epic CRDB-889

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.