cockroachdb / cockroachdb/cockroach

PCR: behavior when performing fast cutback from multiple standbys is possibly incorrect

Open
#131,947 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-master C-bug O-support P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Please describe the issue you observed, and any steps we can take to reproduce it:

Consider the following scenario with primary cluster A, with two standbys B and C:

1. Start PCR from cluster A to two clusters B and C
2. Complete cutover on clusters B and C
3. Start PCR from B back to A using fast cutback, and complete cutover

At this point, what happens if we stop the service on A again and then attempt to start PCR from C back to A using fast cutback? Cluster A gets rewound to the point where cluster C ran cutover and then consumes changes from there.

However, after chatting with @dt, he thinks it's likely that this rewind may not quite be done correctly. David detailed the following sequence events:
- If B cutover at time T5 before C cutover at time T8, when we made A a standby of B, we rewound it to the time B diverged -- T5. This destroys all history in A newer than T5.
- Then we get new history from B, at times greater than T5. So let's say B sends us a row at T7.
- Then we promote A at T10 -- so it still has B's T7 row. Then we stand it back down and tell it to be a secondary to C, it'll rewind to do that, but I wonder what time we rewind to
- It needs to be T5 since that's the last time A has that is in common with C, but I am questioning if we thought of this or if the code is gonna be silly and pick T8, when C diverged

Some testing/digging is the next step here.

**To Reproduce**

Steps to set up the scenario are detailed above.

**Environment:**
- CockroachDB version [e.g. 2.0.x]: 24.1
- Server OS: [e.g. Linux/Distrib]: n/a
- Client app [e.g. `cockroach sql`, JDBC, ...]: n/a

**Additional context**
n/a

Jira issue: CRDB-42759

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.