UnsupportedOperationException when querying a converted dictionary -> noDictionary column
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
We got to this error by
- initially creating a table and ingestion GB/TBs of data
- months later, we added a Json column and reloaded all segments
- months after that, we added the Json column to `noDictionaryColumns`
Now, any queries that hit that json column on segments that were created before the column existed see
```
QueryExecutionError:
java.lang.UnsupportedOperationException
at org.apache.pinot.segment.spi.index.reader.ForwardIndexReader.getInt(ForwardIndexReader.java:372)
at org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValues(DataFetcher.java:576)
at org.apache.pinot.core.common.DataFetcher.fetchStringValues(DataFetcher.java:239)
at org.apache.pinot.core.common.DataBlockCache.getStringValuesForSVColumn(DataBlockCache.java:277)
```
If you query the table with `where is NOT NULL`, then the queries do succeed. So there is at least a workaround.
This cluster is on a version of master from May 2023. I didn't see any issues/fixes related searching for elements in this stack trace.
Contributor guide
Research direction
Start by reproducing the failure with an older segment, a later-added Json column, and that column in noDictionaryColumns. Read ForwardIndexReader.java around getInt, then follow DataFetcher.java and DataBlockCache.java through the stack trace. Done means queries against the affected older segments no longer raise UnsupportedOperationException; compare with the reported IS NOT NULL workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100