[CURATOR-622] Add Randomness to LeaderLatch Elections

Currently, LeaderLatch uses EPHEMERAL_SEQUENTIAL nodes, with the next leader chosen by the lowest numbered node. In a multi-server environment where each server is a participant in multiple elections, the result is that the leader will always be the server that has been up the longest.(Or first to be restarted during a rolling restart)

Instead of using sequentially numbered nodes, I propose instead that the node number for a new participant be created by adding a random number(From a constrained range) to the current leader number.(Defaults to zero) If a node with that number exists, repeat until an available node is found. After initial node creation, all other aspects of the leader election will remain unchanged.

I have an implementation for this that I am testing locally and will submit a PR once the tests are complete.

---
Originally reported by FreemanB, imported from: Add Randomness to LeaderLatch Elections

• status: Open
  
• priority: Major
  
• resolution: Unresolved
  
• imported: 2025-01-21