Cluster stuck with no leader after DR changes all node IPs (Raft references old addresses)

Description

During our Disaster Recovery (DR) process, all Typesense nodes change their IP addresses (e.g., from x.x.26.x to x.x.27.x). Once in the DR environment, we stop the Typesense service on each node and modify only the .ini file (specifically the nodes setting) to reference hostnames. However, Raft persists the old IPs in its logs/snapshots, causing the cluster to remain in a no-leader state after restart.

---

Steps to Reproduce

1. Set up a multi-node cluster (e.g., 3 nodes in x.x.26.x) in the primary data center, each referencing the others in the nodes config.
2. Trigger a DR failover so that all nodes come online in x.x.27.x.
3. Stop the service on each node and edit the .ini to reference the new hostnames (or IPs).
4. Restart the service with the updated .ini. Despite this, the Raft snapshots/logs still reference the old IPs, leaving the cluster without a leader.

---

Expected Behavior

• That updating nodes in the .ini and restarting would allow the cluster to seamlessly transition to the new IP/hostname environment.
• A supported method for Raft to remove or replace the old IP references when all nodes have changed addresses.

---

Actual Behavior

• Even after editing .ini, Raft keeps the old x.x.26.x IP addresses in its internal state.
• Leader elections fail, and the cluster remains stuck with messages like:

  WYYYYMMDD HH:MM:SS raft_server.cpp:734] Multi-node with no leader: refusing to reset peers.
  WYYYYMMDD HH:MM:SS node.cpp:1589] node default_group:x.x.27.x:8107:443 can't do pre_vote
    as it is not in x.x.26.11:8107:443, x.x.26.12:8107:443, x.x.26.13:8107:443
  

---

Troubleshooting Attempts

1. Removing/Editing Raft files (e.g., raft_meta, log_inprogress_...):
   • Often leads to data inconsistencies or collection loss because the Raft state no longer matches the on-disk data.
2. Switching to hostnames in .ini:
   • Does not help if the Raft log was already populated with old IP references, as the log seems to override the new configs.
3. Partial re-bootstrapping:
   • Requires carefully stopping and restarting nodes in a new configuration, which risks data integrity if not done meticulously.

---

Current Workaround

Our current workaround after DR is:

1. Stop the service on each node.
2. Change nodes in each node’s .ini to list only the local hostname.
3. Start each node, creating three separate single-node clusters.
4. Stop the service again on each node.
5. Re-edit nodes to include all three hostnames (the full cluster) in each node’s .ini.
6. Start the service again.

This procedure brings the cluster back, but it is cumbersome and seems more like a workaround than a robust solution.

---

Notes / Request for Guidance

While this process works for now, it feels like a temporary fix for DR scenarios. We would greatly appreciate any official guidance or more robust methods to handle full IP/hostname changes without multiple restarts or risking data loss. If there are other ideas, commands, or best practices for updating or cleaning up Raft references in a safer manner, we are open to suggestions.

Thank you for your assistance and for all your work on Typesense!

Start with the reported messages in raft_server.cpp:734 and node.cpp:1589, then inspect the raft_meta and log_inprogress files mentioned in the troubleshooting notes. Trace how persisted Raft peer addresses interact with the nodes setting during restart. Done should be a supported, data-safe recovery or reconfiguration path for changing every node's address, with coverage for the described DR scenario.