planetscale / planetscale/vitess-operator

Init shard primary flow both in vitess-operator and vtorc lead to primary-replica replication abnormal

Open
#634 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
372
Forks
102
Avg merge
3d 5h
Merged PRs (30d)
15

Description

Overview of the issue

using vitess-operatro to create a vitess instance, with keysapce customer of two shards(Figure 1).
for shard customer:-80, the primary is zone1-2289928654, the replica is zone1-0120139806, the replica can not be replicated,
because it counter duplicate errors, seen as Figure 2.

Replica logs is lost, We can see the primary logs in Figure 3. It shows that the primary-replica relationship is constructed before time 16:11:58,
and then the primary receives the ResetReplication command at 16:12:02, this command is probablly from vitess-operator, code can be seen in Figure 4.

Reproduction Steps

The scenario maybe like this:

  1. operator find no primary
  2. vtorc find no primay
  3. vtorc lock shard ,and select a primary, construcet primary-replica relationship, and then unlock shard;
  4. operator lock shard, and then select a primary, and reset replication, re-construct the primary-replica relationship. Unfortunatly,
    the primary reset master is failed(Figure 5), the binlog file is not purged while the replica relag log and gtid_excuted, gitd_purged is cleared.
    So the replica's has the row (primary key is 172984391830416639" , and it recevies the binglog from gtid:b3767c83-92a8-11ef-add6-066fd3332dcc:23, it reports duplicate primary. If the primary reset master scucuess, and the primary binlog file is cleared, the primary-replica relationship maybe work well.

The problems is that init shard primary is invoked twice, once in vtorc, the second time is in vitess-opearot. Normally, both works well, but
in the sceanrio like above, it breaks the primary-replica replication.

Figure 1(togology)
Image

Figure 2 (replica replication abnormal)
Image

Figure 3 (primary log)
Image

Figure 4 (vitess-operator code)
Image

Figure 5 (replicat duplicat primay key )
Image

Binary Version

Vitess v18.0.1
latest tag vitess operator

Operating System and Environment details

K8S cluster

Log Fragments

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Vitess Operator and vtorc primary-initialization flows described in the reproduction steps, focusing on when each locks the shard, selects a primary, resets replication, and reconstructs relationships. Compare the operator code shown in Figure 4 with the primary logs in Figure 3 and replication evidence in Figures 2 and 5. Done means the duplicate initialization no longer leaves primary-replica replication abnormal in this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.