cockroachdb / cockroachdb/cockroach

crosscluster/logical: in DLQ entry, surface if conflict occurred on local write or replicated write

Open
#155,490 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, a customer has to go through a [manual process](https://cockroachlabs.atlassian.net/wiki/spaces/CKB/pages/4084465707/Playbook+Logical+Data+Replication#Dead-Letter-Queue-Management) to understand if replicating row ended up in the DLQ due to a conflict with a row also written by LDR or from a foreground workload on the destination.

We should annotate the error in the dlq entry write path (code[ ref)](https://github.com/msbutler/cockroach/blob/master/pkg/crosscluster/logical/dead_letter_queue.go#L198) to indicate if the conflicting row has an originTimestamp (from LDR, rare) or not (foreground workload, expected). We could add a new column to the DLQ table too, but maybe thats more complicated than this needs to be.

For this to work nicely, i think we need to teach constraint violation error handling to surface the conflicting row's origin timestamp.

Jira issue: CRDB-55456

Contributor guide

Open the contributing guide

Research direction

Start at pkg/crosscluster/logical/dead_letter_queue.go around line 198, then trace the constraint-violation error handling that produces the DLQ entry. Determine how to expose the conflicting row's originTimestamp and distinguish replicated from foreground conflicts; done means the DLQ error identifies which kind of write caused the conflict.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.