Feature Request: Peer Discovery

Description

I would like to deploy my search service on AWS ECS with the ability to dynamically scale the service without having to restart the service when scaling. The following properties are important to me:

• Nodes can be spawned and destroyed on demand.
• The service can be completely stopped when traffic is nonexistent.
• New nodes can broadcast their existence to join a cluster.

This would make configuring highly-available clusters much simpler, as there doesn't need to be any automation or scripting to ensure that all nodes receive their peering IP addresses as a list.

Can this be done? I'm considering allocating some time for this and I was wondering if this is feasible. I think the biggest point of contention would be the authentication of peers and how to prevent rogue/malicious peers from joining the cluster.

Expected Behavior

• I spawn an instance of TypeSearch with a peering subnet, a peering address, a peering port, and auto-peer discovery.
• The TypeSearch instance sends a multicast to determine if other nodes are available.
• The TypeSearch instance receives no responses and elects itself a leader.
• I spawn 2 instances of TypeSearch within the peering subnet with the same configuration.
• The two added instances send a multicast to find all available instances.
• Each instance acknowledges each others existence and authenticate each other.
• The three instances are now in cluster formation after electing a leader.

Actual Behavior

• I spawn instances of TypeSearch with a list of all nodes that should be considered peers.
• The TypeSense nodes form a cluster.
• I kick off an edge function to scale the service.
• The edge function creates instances of TypeSearch under the same configuration.
• The edge function gathers the IP addresses of all instances and reconfigures the nodes configuration property.
• The edge function restarts all nodes.

Metadata

Typesense Version: Latest
OS: Ubuntu Linux 22.04

Related to:

• #472

The issue names no files, tests, or entry points; start by reviewing the existing peering configuration and related issue #472. Done would mean instances can discover and authenticate peers dynamically, elect a leader, and form a cluster without manually updating nodes or restarting services.