IndexOutOfBoundsException when generating REALTIME inverted index
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
After migrating from `0.8.0` to `0.9.2` we started to see the receive the following exception for some of our high volume REALTIME tables:
```
java.lang.IndexOutOfBoundsException: Index 317 out of bounds for length 151
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeInvertedIndexReader.add(RealtimeInvertedIndexReader.java:59) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:580) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:491) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:524) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
Another similar example:
```
java.lang.IndexOutOfBoundsException: Index 3021 out of bounds for length 27
```
Digging into the stack trace a bit this [method](https://github.com/apache/pinot/blob/f4e55182a8c552c8493d1693868b0c752b1a1d56/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java#L47-L61) appears to operate on the assumption that the `dictId` is either the next index in the underlying `ArrayList` and should be added, or it's already a valid index and could be used to `get(dictId)`.
When this assumption is not met then an `IndexOutOfBoundsException` is thrown and the table stops consuming from that partition.
Note: No table configurations were made between the version upgrades.
Contributor guide
Research direction
Read pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java, especially add, then trace its callers in MutableSegmentImpl.addNewRow and index. Reproduce with a high-volume REALTIME table using the reported stack trace as the starting point. Done means ingestion no longer throws IndexOutOfBoundsException or stops consuming the affected partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100