apache / apache/pinot

Failed to query string column in noDictionaryColumns

Open
#11,704 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

A simple select * query leads to query failure with following. This happens only for a few rows and only in offline segments. The column is single value string with raw forward encoding
```
...
...
"noDictionaryColumns": [
"compressed_data_string"
]
},
"fieldConfigList": [
{
"name": "compressed_data_string",
"encodingType": "RAW",
"indexTypes": [],
"properties": {
"deriveNumDocsPerChunkForRawIndex": "true",
"rawIndexWriterVersion": "3"
}
}
]
```

Error trace:
```
Caught exception while processing query
java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:112)
at org.xerial.snappy.SnappyNative.rawUncompress(SnappyNative.java)
at org.xerial.snappy.Snappy.uncompress(Snappy.java:551)
at o.a.p.s.l.i.c.SnappyDecompressor.decompress(SnappyDecompressor.java:40)
at o.a.p.s.l.s.i.r.f.BaseChunkForwardIndexReader.decompressChunk(BaseChunkForwardIndexReader.java:140)
... 25 common frames omitted
Wrapped by: java.lang.RuntimeException: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at o.a.p.s.l.s.i.r.f.BaseChunkForwardIndexReader.decompressChunk(BaseChunkForwardIndexReader.java:143)
at o.a.p.s.l.s.i.r.f.BaseChunkForwardIndexReader.getChunkBuffer(BaseChunkForwardIndexReader.java:121)
at o.a.p.s.l.s.i.r.f.VarByteChunkSVForwardIndexReader.getStringCompressed(VarByteChunkSVForwardIndexReader.java:79)
at o.a.p.s.l.s.i.r.f.VarByteChunkSVForwardIndexReader.getString(VarByteChunkSVForwardIndexReader.java:68)
at o.a.p.s.l.s.i.r.f.VarByteCh...
```

Contributor guide

Open the contributing guide

Research direction

Start with BaseChunkForwardIndexReader.decompressChunk and getChunkBuffer, then inspect VarByteChunkSVForwardIndexReader.getStringCompressed and the SnappyDecompressor path shown in the trace. Reproduce a simple select * query against an offline segment containing the RAW single-value string column and determine why only some rows fail; done means the query no longer raises FAILED_TO_UNCOMPRESS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.