apache / apache/pinot

IndexOutOfBoundsException when generating REALTIME inverted index

Open
#7,923 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

After migrating from `0.8.0` to `0.9.2` we started to see the receive the following exception for some of our high volume REALTIME tables:

```
java.lang.IndexOutOfBoundsException: Index 317 out of bounds for length 151
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeInvertedIndexReader.add(RealtimeInvertedIndexReader.java:59) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:580) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:491) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:524) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at java.lang.Thread.run(Thread.java:829) [?:?]
```

Another similar example:
```
java.lang.IndexOutOfBoundsException: Index 3021 out of bounds for length 27
```

Digging into the stack trace a bit this [method](https://github.com/apache/pinot/blob/f4e55182a8c552c8493d1693868b0c752b1a1d56/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java#L47-L61) appears to operate on the assumption that the `dictId` is either the next index in the underlying `ArrayList` and should be added, or it's already a valid index and could be used to `get(dictId)`.

When this assumption is not met then an `IndexOutOfBoundsException` is thrown and the table stops consuming from that partition.

Note: No table configurations were made between the version upgrades.

Contributor guide

Open the contributing guide

Research direction

Read pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java, especially add, then trace its callers in MutableSegmentImpl.addNewRow and index. Reproduce with a high-volume REALTIME table using the reported stack trace as the starting point. Done means ingestion no longer throws IndexOutOfBoundsException or stops consuming the affected partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.