Possible invalid read issue with GroupBy V2 spilled dictionaries
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.13.0-incubating
### Description
A user reported seeing issues when using a GroupBy V2 query: https://groups.google.com/forum/#!topic/druid-user/uM690lhVo7k
```
2019-05-08T21:16:35,187 ERROR [qtp269685385-140[groupBy_[(redacted)]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.fasterxml.jackson.databind.RuntimeJsonMappingException, exceptionMessage=Can not deserialize instance of java.lang.String out of VALUE_NULL token
at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], exception=com.fasterxml.jackson.databind.RuntimeJsonMappingException: Cannot deserialize instance of java.lang.String out of VALUE_NULL token at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], query=GroupByQuery{dataSource='(redacted)', querySegmentSpec=MultipleSpecificSegmentSpec{descriptors=[SegmentDescriptor{interval=2019-05-01T00:00:00.000Z/2019-05-05T00:00:00.000Z, version='2019-05-08T12:01:15.823Z', partitionNumber=0}]}, virtualColumns=[], limitSpec=NoopLimitSpec, dimFilter=((redacted)}
com.fasterxml.jackson.databind.RuntimeJsonMappingException: Can not deserialize instance of java.lang.String out of VALUE_NULL token
at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259]
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:194) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.mergeAndGetDictionary(SpillingGrouper.java:223) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.tryMergeDictionary(ConcurrentGrouper.java:392) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.iterator(ConcurrentGrouper.java:320) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.(CloseableGrouperIterator.java:44) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:426) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:414) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:282) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:158) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:64) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80) ~[druid-common-0.13.0-incubating.jar:0.13.0-incubating]
```
From the stack trace, it looks like invalid data is possibly being read/written from the on-disk spilled dictionaries.
The error is coming from:
```
while (dictIterator.hasNext()) {
mergedDictionary.add(dictIterator.next());
}
```
Based on the user report, the issue is possibly related to amount of query results being processed.
Contributor guide
Research direction
Start with org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.mergeAndGetDictionary and the callers in ConcurrentGrouper and RowBasedGrouperHelper. Reproduce a GroupBy V2 query that processes a large result set and spills dictionaries, then verify that the spilled data can be read without the null String deserialization error and add regression coverage if an existing test location is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100