cockroachdb / cockroachdb/cockroach
PCR: behavior when performing fast cutback from multiple standbys is possibly incorrect
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Please describe the issue you observed, and any steps we can take to reproduce it:
Consider the following scenario with primary cluster A, with two standbys B and C:
1. Start PCR from cluster A to two clusters B and C
2. Complete cutover on clusters B and C
3. Start PCR from B back to A using fast cutback, and complete cutover
At this point, what happens if we stop the service on A again and then attempt to start PCR from C back to A using fast cutback? Cluster A gets rewound to the point where cluster C ran cutover and then consumes changes from there.
However, after chatting with @dt, he thinks it's likely that this rewind may not quite be done correctly. David detailed the following sequence events:
- If B cutover at time T5 before C cutover at time T8, when we made A a standby of B, we rewound it to the time B diverged -- T5. This destroys all history in A newer than T5.
- Then we get new history from B, at times greater than T5. So let's say B sends us a row at T7.
- Then we promote A at T10 -- so it still has B's T7 row. Then we stand it back down and tell it to be a secondary to C, it'll rewind to do that, but I wonder what time we rewind to
- It needs to be T5 since that's the last time A has that is in common with C, but I am questioning if we thought of this or if the code is gonna be silly and pick T8, when C diverged
Some testing/digging is the next step here.
**To Reproduce**
Steps to set up the scenario are detailed above.
**Environment:**
- CockroachDB version [e.g. 2.0.x]: 24.1
- Server OS: [e.g. Linux/Distrib]: n/a
- Client app [e.g. `cockroach sql`, JDBC, ...]: n/a
**Additional context**
n/a
Jira issue: CRDB-42759
Contributor guide
Assessment
This issue has not been assessed yet.