Support "competing" readers

Currently all readers read the entire log and there is no mechanism for "competing" reads, i.e. where multiple readers read entries/chunks in round-robin order, i.e. in order to increase the speed at which entries for a given stream are processed.

A possible design for doing competing reads:

• A competing read coordinator (CRC) process is co-hosted with the osiris writer (leader)
• The CRC acts like a reader although it only ever scans the chunk headers.
• Readers that want to "compete" (collaborate may be a better term) over a log register with the CRC and wait for the CRC to inform them of the next chunk id (offset) to read from.
• The CRC will thus allocate chunk ids to competing reads and maintain state of the current readers, what chunk ids they have been allocated.
• Readers will ack back when they are finished processing a chunk id and so that they can be allocated another chunk id (some degree of pipelining should be be allowed).
• The CRC persists the current read state in the log as a special entry type so that it can be replicated recovered from anywhere
• Thus chunk ids are allocated to available readers in a round-robin-ish manner allowing.
• Although allocation of chunk ids happen on the writer node the reads can do the actual reads on replica nodes which will further scale out reads across the cluster.

Downsides:

• Ordering is gone as if a reader fails whilst being allocated a chunk id this chunk id needs to be given to an existing reader which may already have processed a higher chunk id, resulting in this reader having read chunks out of order. That said for competing consumes this is always the case.

The issue proposes a competing-read coordinator co-hosted with the osiris writer, but names no files, tests, or concrete entry points. Start by locating the writer and reader processes and determine where chunk allocation and persisted read state belong. Done would require an implemented, recoverable competing-read design with reader acknowledgements and replica reads.