apache / apache/druid

Possible invalid read issue with GroupBy V2 spilled dictionaries

Open
#7,690 5 comments 1 reaction 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.13.0-incubating

### Description

A user reported seeing issues when using a GroupBy V2 query: https://groups.google.com/forum/#!topic/druid-user/uM690lhVo7k

```
2019-05-08T21:16:35,187 ERROR [qtp269685385-140[groupBy_[(redacted)]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.fasterxml.jackson.databind.RuntimeJsonMappingException, exceptionMessage=Can not deserialize instance of java.lang.String out of VALUE_NULL token
at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], exception=com.fasterxml.jackson.databind.RuntimeJsonMappingException: Cannot deserialize instance of java.lang.String out of VALUE_NULL token at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], query=GroupByQuery{dataSource='(redacted)', querySegmentSpec=MultipleSpecificSegmentSpec{descriptors=[SegmentDescriptor{interval=2019-05-01T00:00:00.000Z/2019-05-05T00:00:00.000Z, version='2019-05-08T12:01:15.823Z', partitionNumber=0}]}, virtualColumns=[], limitSpec=NoopLimitSpec, dimFilter=((redacted)}
com.fasterxml.jackson.databind.RuntimeJsonMappingException: Can not deserialize instance of java.lang.String out of VALUE_NULL token
at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259]
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:194) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.mergeAndGetDictionary(SpillingGrouper.java:223) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.tryMergeDictionary(ConcurrentGrouper.java:392) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.iterator(ConcurrentGrouper.java:320) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.(CloseableGrouperIterator.java:44) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:426) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:414) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:282) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:158) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:64) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80) ~[druid-common-0.13.0-incubating.jar:0.13.0-incubating]
```

From the stack trace, it looks like invalid data is possibly being read/written from the on-disk spilled dictionaries.

The error is coming from:
```
while (dictIterator.hasNext()) {
mergedDictionary.add(dictIterator.next());
}
```

Based on the user report, the issue is possibly related to amount of query results being processed.

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.mergeAndGetDictionary and the callers in ConcurrentGrouper and RowBasedGrouperHelper. Reproduce a GroupBy V2 query that processes a large result set and spills dictionaries, then verify that the spilled data can be read without the null String deserialization error and add regression coverage if an existing test location is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.