SIGSEGV errors while ingesting realtime data

While ingesting real-time data into a single table we are periodically getting SIGSEGV fatal errors on all servers.

Context:
- Pinot 1.0.0 using MMAP loading

Table is composed of:
- a few dimension fields (SV with inverted indexes)
- a lot of metric fields (SV with forward index only)
- a timestamp field

Below is a short extract of the core dump log (full one is attached).

```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f9ba3e89056, pid=1, tid=247
#
# JRE version: OpenJDK Runtime Environment Corretto-11.0.20.9.1 (11.0.20.1+9) (build 11.0.20.1+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.20.9.1 (11.0.20.1+9-LTS, mixed mode, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136]
#
# Core dump will be written. Default location: /opt/pinot/core.1
#
# If you would like to submit a bug report, please visit:
# https://github.com/corretto/corretto-11/issues/
#

--------------- S U M M A R Y ------------

Command Line: -Xms512M -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ErrorFileToStdout -XX:-TieredCompilation -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-server-log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dplugins.dir=/opt/pinot/plugins -Dapp.name=pinot-admin -Dapp.pid=1 -Dapp.repo=/opt/pinot/lib -Dapp.home=/opt/pinot -Dbasedir=/opt/pinot org.apache.pinot.tools.admin.PinotAdministrator StartServer -clusterName pinot -zkAddress pinot-zookeeper:2181 -configFileName /var/pinot/server/config/pinot-server.conf

Host: Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz, 16 cores, 32G, Amazon Linux release 2 (Karoo)
Time: Tue Feb 6 12:41:51 2024 UTC elapsed time: 2970.773692 seconds (0d 0h 49m 30s)

--------------- T H R E A D ---------------

Current thread (0x00007f9a04062800): JavaThread "obf_50d04beb9d1d306c5a5e45167656a565abc036da0763183eb9f074402fbd4f2c__0__3__20240206T1214Z" daemon [_thread_in_Java, id=247, stack(0x00007f9a264f6000,0x00007f9a265f7000)]

Stack: [0x00007f9a264f6000,0x00007f9a265f7000], sp=0x00007f9a265f55e0, free space=1021k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136]
J 3819 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.getRecord(ILorg/apache/pinot/spi/data/readers/GenericRow;)V (106 bytes) @ 0x00007f9ba3e726ac [0x00007f9ba3e72080+0x000000000000062c]
J 4599 c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build()V (376 bytes) @ 0x00007f9ba3dc04c4 [0x00007f9ba3dbfc60+0x0000000000000864]
j org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(Lorg/apache/pinot/segment/spi/creator/SegmentVersion;Lorg/apache/pinot/common/metrics/ServerMetrics;)V+282
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentInternal(Z)Lorg/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager$SegmentBuildDescriptor;+206
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentAndReplace()Z+2
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run()V+561
j java.lang.Thread.run()V+11 java.base@11.0.20.1
v ~StubRoutines::call_stub
V [libjvm.so+0x8e13bb] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x39b
V [libjvm.so+0x8df37d] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1ed
V [libjvm.so+0x98ae7c] thread_entry(JavaThread*, Thread*)+0x6c
V [libjvm.so+0xedf730] JavaThread::run()+0x280
V [libjvm.so+0xedc0ff] Thread::call_run()+0x14f
V [libjvm.so+0xc78ea6] thread_native_entry(Thread*)+0xe6

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f95f0137ac8
```

> - Sensitive data has been obfuscated.
> - Tiered Compilation was disabled to try fixing this kind of SIGSEGV errors while invoking MutableForwardIndex / OffHeapMutableDictionary

See full core dump here:
[SIGSEGV-obf.log](https://github.com/apache/pinot/files/14183217/SIGSEGV-obf.log)

How to investigate deeper ?

Start with the attached SIGSEGV log and the stack frames in PinotSegmentColumnReader.getValue, PinotSegmentRecordReader.getRecord, and RealtimeSegmentConverter.build. Reproduce realtime MMAP ingestion with the described table shape, then trace the MutableForwardIndex and OffHeapMutableDictionary path. Done means the crash cause is identified and a verified fix or mitigation prevents the SIGSEGV.

Written by the indexing model from the issue text.