tarantool / tarantool/tarantool

Error converting anonymous replica to regular

Open
#12,603 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Lua
Stars
3.7k
Forks
419
Avg merge
1d 23h
Merged PRs (30d)
88

Description

Bug description

We added an anonymous replica to the replica set as a workaround for a known issue where adding regular replicas fails in a cluster with synchronous transactions enabled. When attempting to convert this anonymous replica to a regular replica, the operation failed with errors.

tarantool version | 3.4.1-0-g096322fad

Steps to reproduce

Start anon replicas with next config
replication:
anon: true
election_mode: off

Reconfigure replicas
replication:
anon: false
election_mode: voter

WA and Example:
Procedure for removing old replica and adding anonymous replica

  1. Enable automatic expulsion in the global configuration
    Set the following parameters under replication at the global level:
replication:
  autoexpel:
    by: prefix
    enabled: true
    prefix: 'your_prefix'
  1. Remove old replica

  2. Remove old replica from config

  3. Deploying new replica with next configuration

replication:
  anon: true
  election_mode: off
  1. Explicitly specify replication peers for all existing nodes
    Example for the first candidate node:
          tdb_storage-1-2:
            iproto:
              advertise:
                client: 10.181.64.13:3404
              listen:
              - uri: 10.181.64.13:3404
            memtx:
              memory: 536870912
            replication:
              election_mode: candidate
              peers:
                - replicator:i-changed-a-password-here@10.181.8.1:3403 (replica candidate)
                - replicator:i-changed-a-password-here@10.181.64.13:3404 (master candidate)
            roles_cfg:
              roles.metrics-export:
                http:
                - endpoints:
                  - format: prometheus
                    path: /metrics
                  listen: 8404

Example for the second candidate node

          tdb_storage-1-1:
            iproto:
              advertise:
                client: 10.181.8.1:3403
              listen:
              - uri: 10.181.8.1:3403
            memtx:
              memory: 536870912
            replication:
              election_mode: candidate
              peers:
               - replicator:i-changed-a-password-here@10.181.8.1:3403
               - replicator:i-changed-a-password-here@10.181.64.13:3404 
            roles_cfg:
              roles.metrics-export:
                http:
                - endpoints:
                  - format: prometheus
                    path: /metrics
                  listen: 8403
  1. Explicitly define the UUID for the new replica in its configuration
    Example for the new replica:
          tdb_storage-1-3:
            iproto:
              advertise:
                client: 10.181.64.13:3405
              listen:
              - uri: 10.181.64.13:3405
            database:
              instance_uuid: 1c32c157-7bb7-4fd6-94ce-de5a97cab588
            memtx:
              memory: 536870912
            replication:
              anon: true
              election_mode: off
            roles_cfg:
              roles.metrics-export:
                http:
                - endpoints:
                  - format: prometheus
                    path: /metrics
                  listen: 8405
  1. Change the parameters of the added replica
    Update configuration to:
replication:
  anon: false
  election_mode: voter
  1. Fix the missing instance name warning box_cfg.apply: name tdb_storage-1-3 for b7e7e85a-e2cc-40d0-8cb1-a8454ec5919d uuid is missing from the snapshot. It will be automatically set when possible.
10.181.64.13:3404> box.space._cluster:fselect()
---
- |-
  +-----+--------------------------------------+-----------------+
  | id  |                 uuid                 |      name       |
  +-----+--------------------------------------+-----------------+
  |  1  |"8a07b2ee-04af-4d67-bddd-b797bf1f1693"|"tdb_storage-1-2"|
  |  2  |"b7e7e85a-e2cc-40d0-8cb1-a8454ec5919d"|                 |
  |  3  |"dd33fa06-d06f-4146-a5cf-226a201d4f01"|"tdb_storage-1-1"|
  +-----+--------------------------------------+-----------------+
...

10.181.64.13:3404> box.space._cluster:update(2, {{'=', 'name', 'tdb_storage-1-
3'}})
---
- [2, 'b7e7e85a-e2cc-40d0-8cb1-a8454ec5919d', 'tdb_storage-1-3']
...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is identified. Start by reproducing the anonymous-to-regular replica transition with the configurations and cluster setup described, then inspect the replication reconfiguration path and the _cluster name update shown in the report. Done means the replica becomes a voter without errors or a missing-name warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.