apache / apache/pinot

UnsupportedOperationException when querying a converted dictionary -> noDictionary column

Open
#11,893 8 comments 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

We got to this error by
- initially creating a table and ingestion GB/TBs of data
- months later, we added a Json column and reloaded all segments
- months after that, we added the Json column to `noDictionaryColumns`

Now, any queries that hit that json column on segments that were created before the column existed see
```
QueryExecutionError:
java.lang.UnsupportedOperationException
at org.apache.pinot.segment.spi.index.reader.ForwardIndexReader.getInt(ForwardIndexReader.java:372)
at org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValues(DataFetcher.java:576)
at org.apache.pinot.core.common.DataFetcher.fetchStringValues(DataFetcher.java:239)
at org.apache.pinot.core.common.DataBlockCache.getStringValuesForSVColumn(DataBlockCache.java:277)
```

If you query the table with `where is NOT NULL`, then the queries do succeed. So there is at least a workaround.

This cluster is on a version of master from May 2023. I didn't see any issues/fixes related searching for elements in this stack trace.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with an older segment, a later-added Json column, and that column in noDictionaryColumns. Read ForwardIndexReader.java around getInt, then follow DataFetcher.java and DataBlockCache.java through the stack trace. Done means queries against the affected older segments no longer raise UnsupportedOperationException; compare with the reported IS NOT NULL workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.