shenandoah GC can cause "ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum" [LUCENE-5899]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
User report of bizare ClassCastException when running some lucene code with the (experimental) shenandoah GC
```Java
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum
at org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:127)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:110)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:399)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4163)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
```
```Java
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Invalid vLong detected (negative values disallowed)
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.io.IOException: Invalid vLong detected (negative values disallowed)
at org.apache.lucene.store.DataInput.readVLong(DataInput.java:193)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.(BlockTreeTermsReader.java:169)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:197)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:107)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:668)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
```
```Java
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.NullPointerException
at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.(BlockTreeTermsReader.java:169)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:197)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:107)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:668)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.NullPointerException
at org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.reset(MappingMultiDocsAndPositionsEnum.java:50)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:188)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:399)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4163)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
```
---
Migrated from [LUCENE-5899](https://issues.apache.org/jira/browse/LUCENE-5899) by Littlestar, updated Aug 28 2014
Environment:
```
I test lucene on shenandoah with big memory + bigdata(32g, 128g).
http://openjdk.java.net/jeps/189
http://icedtea.classpath.org/hg/shenandoah/
```
Contributor guide
Research direction
Start by reviewing the merge paths named in the traces, especially PostingsConsumer.merge, TermsConsumer.merge, SegmentMerger.mergeTerms, and the MappingMultiDocs* classes. Reproduce the failures with the reported Shenandoah GC environment and large-memory workload, then use the stack traces to determine the cause and verify that merges no longer produce the reported ClassCastException, invalid vLong, or NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100