apache / apache/lucene

IllegalStateException: wrong number of values added during doc values merge [LUCENE-8823]

Open
#9,867 0 comments 0 reactions 0 assignees View on GitHub
affects-version:7.6 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Here's another mysterious exception we hit in production, on Lucene 7.x snapshot release (near 7.6), OpenJDK 11:
```
2019-05-31T05:49:22,443 [ERROR] (Lucene Merge Thread #0) com.amazon.lucene.util.UncaughtExceptionHandler: Uncaught exception: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784 in thread Thread[Lucene Merge Thread #0,5,main] org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: java.lang.IllegalStateException: Wrong number of values added, expected: 97006, got: 95784
at org.apache.lucene.util.packed.DirectWriter.finish(DirectWriter.java:94)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValuesSingleBlock(Lucene70DocValuesConsumer.java:283)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:263)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addNumericField(Lucene70DocValuesConsumer.java:110)
at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:175)
at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:126)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4438)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4060)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at com.amazon.lucene.index.ConcurrentMergeSchedulerWrapper.doMerge(ConcurrentMergeSchedulerWrapper.java:54)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
```
Merging of a numeric doc values field failed because too few values were added.  This may also be a JVM bug, though our doc values codec code is quite complex so it could also be a Lucene bug!

---
Migrated from [LUCENE-8823](https://issues.apache.org/jira/browse/LUCENE-8823) by Michael McCandless (@mikemccand)

Contributor guide

Open the contributing guide

Research direction

Start with DirectWriter.finish and the Lucene70DocValuesConsumer.writeValuesSingleBlock/writeValues path shown in the stack trace, then follow DocValuesConsumer.mergeNumericField during segment merging. Reproduce the numeric doc values merge on the reported Lucene 7.x snapshot and OpenJDK 11 setup; done means identifying whether the value-count mismatch is in Lucene or the JVM and preventing the merge exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.