IllegalStateException: wrong number of values added during doc values merge [LUCENE-8823]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Here's another mysterious exception we hit in production, on Lucene 7.x snapshot release (near 7.6), OpenJDK 11:
```
2019-05-31T05:49:22,443 [ERROR] (Lucene Merge Thread #0) com.amazon.lucene.util.UncaughtExceptionHandler: Uncaught exception: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784 in thread Thread[Lucene Merge Thread #0,5,main] org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784
at org.apache.lucene.util.packed.DirectWriter.finish(DirectWriter.java:94)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValuesSingleBlock(Lucene70DocValuesConsumer.java:283)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:263)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addNumericField(Lucene70DocValuesConsumer.java:110)
at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:175)
at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:126)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4438)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4060)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at com.amazon.lucene.index.ConcurrentMergeSchedulerWrapper.doMerge(ConcurrentMergeSchedulerWrapper.java:54)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
```
Merging of a numeric doc values field failed because too few values were added. This may also be a JVM bug, though our doc values codec code is quite complex so it could also be a Lucene bug!
---
Migrated from [LUCENE-8823](https://issues.apache.org/jira/browse/LUCENE-8823) by Michael McCandless (@mikemccand)
Contributor guide
Research direction
Start with DirectWriter.finish and the Lucene70DocValuesConsumer.writeValuesSingleBlock/writeValues path shown in the stack trace, then follow DocValuesConsumer.mergeNumericField during segment merging. Reproduce the numeric doc values merge on the reported Lucene 7.x snapshot and OpenJDK 11 setup; done means identifying whether the value-count mismatch is in Lucene or the JVM and preventing the merge exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100