cockroachdb / cockroachdb/cockroach

logical: perform validation on source tables before ingestion

Open
#169,243 1 comment 0 reactions 0 assignees View on GitHub
A-cdc A-logical-data-replication C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently LDR performs validation that the destination tables are valid targets for replication via `CheckLogicalReplicationCompatibility`. This validation has a few gaps for the CREATE TABLE flow:

1. Create table has stricter requirements on what constitutes a valid replication target. Consider UDTs, these are supported in the non create table flow, but not in the create table flow as we need to validate the referenced UDTs exist in the replication set and rewrite their IDs. Currently attempting to replicate a table with a UDT will fail with a non specific `cross database type references are not supported` error. Another edge case is VIEW statements, which will silently fail for the same reason.
2. This validation happens after we ingest the source tables and create the destination tables, i.e. after we rewrite IDs. Ideally we would like the rewrite to throw an error if we ever hit an unmapped ID, as this likely means we will have a corrupted descriptor. However because we don't perform validation until after, throwing an error in rewrite would obscure the specific "TRIGGER not supported" errors.

Instead, we should perform validation on the source tables before ingestion. This would allow us to throw explicit errors up front, and add the more general fallback in Rewrite.

Jira issue: CRDB-63348

Contributor guide

Open the contributing guide

Research direction

Trace CheckLogicalReplicationCompatibility and Rewrite from the logical replication ingestion path, focusing on where source tables are available before ingestion. Done means validation reports explicit errors for unsupported UDT, VIEW, and TRIGGER cases before destination creation, while Rewrite provides a general fallback for unmapped IDs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.