Enhancement: persist commit index in LogStore to accelerate recovery

Today I realised my mental model of how our Raft library works is subtly wrong.

I assumed that when a node restarts, it loads the most recent snapshot into the FSM and replays committed logs from disk into the FSM before rejoining the cluster.

In fact, we only load from snapshot since currently we don't persist any information about which logs are known to be committed which means that replaying them all might apply a few uncommitted ones from the end of the log.

In practice this works OK in most cases because if the node is elected leader then it will write a new log which as soon as it's committed will trigger applying all the other committed logs in to the FSM too. If it's a follower it doesn't catch up it's FSM until the first AppendEntries RPC from the leader tells it the current commit index in the cluster.

As we consider reducing snapshot frequency with WAL LogStore being able to store lots more logs performantly, this will become more of an issue because it will mean followers are much more "stale" when they start serving requests.

To fix this correctly, we'd need to persist the last known commit index so that we know which prefix of the logs it's safe to apply on startup. We could periodically persist that to the StableStore but that increases amount of disk transactions and trades that off against how recent the index stays. Currently StableStore performance is not an issue because it's only updated infrequently during elections. This would make it much more critical.

An alternative that would be more efficient but a little more complicate would be:

1. Introduce a new optional interface called something like:

   type CommitTrackingLogStore interface {
       SetCommitIndex(idx uint64) error
       ReadCommitIndex() (uint64, error)
   }
   

2. Have Raft detect that the LogStore also implements this just before calling StoreLogs during replication or leader's dispatchLogs, if it does, call SetCommitIndex first with the most recent commitIndex we know
3. Implementations may store that index in memory until the next StoreLogs call and then write it out on disk along with the log data.
4. When the LogStore recovers it's state on open, it would recover the last persisted commit index to and load it into memory
5. When Raft starts up it can restore snapshot like now then...
6. Check if the LogStore implements CommitTrackingLogStore. If it does, load the commit index from the store and replay all logs from lastSnapshotIndex up to and including the commit index that was persisted, before completing startup.

Start by tracing the LogStore and StableStore interfaces, the replication paths that call StoreLogs, and startup snapshot recovery. Compare the persistence trade-offs described in the issue, then verify that a recovered node restores the persisted commit index and replays only committed logs before startup completes.