Proposal: Use slots in global services

@aluzzardi wrote:

Unrelated to this PR really, but every time we deal with slots vs
node IDs we make a step backwards in decoupling the system (e.g. running
global / replicated outside of core).

In order to decouple, either:

• Core doesn't rely on slots nor node IDs. Today some components deeply rely on
  that such as the reaper, or
• Orchestrators don't use different concepts (everything uses a slot ID or a
  node ID or whatnot)

We've made a bunch of those in the past (e.g. if global then nodeid else slotid) and we're getting the system more tightly coupled each time (and this
PR is no exception).

I propose the following:

• Allocate a SlotID for each node we create. Since slots are defined as uint64 in the protobufs, this can't be the same as the node ID. Two ideas for how to allocate these SlotIDs:
  • Randomly generate a SlotID, then check for uniqueness, and repeat if necessary. We would index the nodes by slot ID to have an efficient check for uniqueness.
  • Use the raft index when the node is created as a SlotID. This is guaranteed to be monotonic, so as long as we only create one node per transaction, we are guaranteed uniqueness without paying the cost of extra indexing on the nodes.
• When the global orchestrator creates a task, it sets Task.Slot to Node.SlotID as well as setting Task.NodeID to Node.ID.
• Gradually, we can remove special casing for global services that have to use node IDs instead of slots. Everything would use slots going forward.

Thoughts?

cc @dongluochen @aluzzardi

Start by tracing node creation, the global orchestrator, and the protobuf definitions for SlotID, Task.Slot, and Task.NodeID. Determine how slots and node IDs are currently assigned and consumed, then assess whether all global-service paths can use slots without breaking reaper or replicated-service behavior.