Reduce ingestion latency during segment sealing

We have some use cases that demand very fresh data. For example, display events aggregates of the **last minute** for decision making.

However, as shown in the diagram below, the ingestion latency spikes during segment sealing, which breaches the SLA

The segment size is 5MB, so in this particular case, it seals every a couple of hours. We could try smaller segment sizes, but that may result in different issues like too many zookeeper nodes.

The solution to this could be more optimistic on new segment creation: the new segments start ingestion without waiting for the sealing. However, this can complicate the state machine, as the segment cleanup is more complicated when the segment commit fails.

Any other thoughts to mitigate this issue?

Start by reviewing the segment-sealing and ingestion state machine, focusing on the latency spike described in the diagram and the proposed optimistic new-segment path. Compare possible mitigations, including smaller segments, and define cleanup behavior when segment commit fails; the issue does not specify an agreed implementation or completion test.

Written by the indexing model from the issue text.