planetscale / planetscale/vitess-operator
Init shard primary flow both in vitess-operator and vtorc lead to primary-replica replication abnormal
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 372
- Forks
- 102
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 15
Description
Overview of the issue
using vitess-operatro to create a vitess instance, with keysapce customer of two shards(Figure 1).
for shard customer:-80, the primary is zone1-2289928654, the replica is zone1-0120139806, the replica can not be replicated,
because it counter duplicate errors, seen as Figure 2.
Replica logs is lost, We can see the primary logs in Figure 3. It shows that the primary-replica relationship is constructed before time 16:11:58,
and then the primary receives the ResetReplication command at 16:12:02, this command is probablly from vitess-operator, code can be seen in Figure 4.
Reproduction Steps
The scenario maybe like this:
- operator find no primary
- vtorc find no primay
- vtorc lock shard ,and select a primary, construcet primary-replica relationship, and then unlock shard;
- operator lock shard, and then select a primary, and reset replication, re-construct the primary-replica relationship. Unfortunatly,
the primary reset master is failed(Figure 5), the binlog file is not purged while the replica relag log and gtid_excuted, gitd_purged is cleared.
So the replica's has the row (primary key is 172984391830416639" , and it recevies the binglog from gtid:b3767c83-92a8-11ef-add6-066fd3332dcc:23, it reports duplicate primary. If the primary reset master scucuess, and the primary binlog file is cleared, the primary-replica relationship maybe work well.
The problems is that init shard primary is invoked twice, once in vtorc, the second time is in vitess-opearot. Normally, both works well, but
in the sceanrio like above, it breaks the primary-replica replication.
Figure 1(togology)
Figure 2 (replica replication abnormal)
Figure 3 (primary log)
Figure 4 (vitess-operator code)
Figure 5 (replicat duplicat primay key )
Binary Version
Vitess v18.0.1
latest tag vitess operator
Operating System and Environment details
K8S cluster
Log Fragments
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Vitess Operator and vtorc primary-initialization flows described in the reproduction steps, focusing on when each locks the shard, selects a primary, resets replication, and reconstructs relationships. Compare the operator code shown in Figure 4 with the primary logs in Figure 3 and replication evidence in Figures 2 and 5. Done means the duplicate initialization no longer leaves primary-replica replication abnormal in this scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100