Lock Contention in Controller During Realtime Segment Commit Events

We are seeing an issue with one of our high ingestion throughput tables where ingestion lag continues to increase because of a table-level lock in Pinot controller.

https://github.com/apache/pinot/blob/723b764bc91275c0b8361d3f9135f151b6404c39/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L594

@Jackie-Jiang I think we briefly discussed this yesterday. I don't have enough context about this, but one potential solution could be to add some jitter in the number of docs in a segment at partition level so these events arrive at slightly different times. But I don't think that is tenable and we may need a more proper fix.

Also, there is a lock being taken in logback which is causing a lot of contention (there were 10 threads blocked in a sample thread dump I took). That seems like a simpler fix which we may wanna do anyways.

```
...
"grizzly-http-server-435" #625 prio=5 os_prio=0 cpu=16853811.42ms elapsed=7356619.65s tid=0x00007f0283f80000 nid=0x2d1 waiting for monitor entry [0x00007edb0c584000]
java.lang.Thread.State: BLOCKED (on object monitor)
at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:151)
- waiting to lock <0x00007ee26209cf80> (a ch.qos.logback.classic.Logger)
at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:53)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.(SegmentCompletionManager.java:437)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.fsmInHolding(SegmentCompletionManager.java:399)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.lookupOrCreateFsm(SegmentCompletionManager.java:162)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentConsumed(SegmentCompletionManager.java:197)
```

Start with PinotLLCRealtimeSegmentManager.java at the linked line and trace the realtime segment commit event path. Use the SegmentCompletionManager stack trace, especially lookupOrCreateFsm and SegmentCompletionFSM initialization, to investigate the controller and Logback contention under high-ingestion load. Done means the contention is addressed and ingestion lag no longer increases during these events.

Written by the indexing model from the issue text.