apache / apache/lucene

Spooky exception merging doc values [LUCENE-8790]

Open
#9,834 0 comments 0 reactions 0 assignees View on GitHub
affects-version:7.5 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We hit this exciting exception; we don't have a test case reproducing it, and staring at the code, I don't see how we can hit a `NullPointerException` on this line:
```
[May 2, 2019, 7:24 PM] Barrowman, Adam: 2019-05-02T18:32:10,561 [ERROR] (Lucene Merge Thread #1) com.amazon.lucene.util.UncaughtExceptionHandler: Uncaught exception: org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException in thread Thread[Lucene Merge Thread #1,5,main] org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: java.lang.NullPointerException
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValuesSingleBlock(Lucene70DocValuesConsumer.java:279)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:263)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedNumericField(Lucene70DocValuesConsumer.java:536)
at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:371)
at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:143)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:126)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4438)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4060)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at com.amazon.lucene.index.ConcurrentMergeSchedulerWrapper.doMerge(ConcurrentMergeSchedulerWrapper.java:54)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
```
It seems like the `encode.get(v)` somehow returned null, which should not happen as long as the values we iterated from the `SortedNumericValues` were the same up above (in `writeValues`) and in `writeValuesSingleBlock`.  Confused...

Note that we are using a 7.x snapshot, so it is possible this was a bug in 7.x at that time, fixed before the next 7.x release though when I compare the affected code against 8.x backwards codec, it looks the same.

---
Migrated from [LUCENE-8790](https://issues.apache.org/jira/browse/LUCENE-8790) by Michael McCandless (@mikemccand)
Environment:
```
We are on a Lucene 7.x snapshot, githash 

935b0c89c6ecb446d7f05d938207760cd64bcd04, using the default Codec, with a static sort.
```

Contributor guide

Open the contributing guide

Research direction

Start at Lucene70DocValuesConsumer.writeValuesSingleBlock on line 279, then trace its callers writeValues and addSortedNumericField. Reproduce the reported NullPointerException during segment merging, ideally with a focused test using the default Codec and static sort; done means the failure is reproduced and the affected merge behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.