UnsupportedOperationException: unused: not a comparsion-based sort during IndexWriter flush [LUCENE-8822]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
We hit this very strange exception in production 7.x snapshot (near 7.6), OpenJDK 11:
```
Caused by: java.lang.UnsupportedOperationException: unused: not a comparison-based sort
at org.apache.lucene.util.MSBRadixSorter.compare(MSBRadixSorter.java:115)
at org.apache.lucene.util.Sorter.siftDown(Sorter.java:235)
at org.apache.lucene.util.Sorter.heapify(Sorter.java:228)
at org.apache.lucene.util.MSBRadixSorter.computeCommonPrefixLengthAndBuildHistogram(MSBRadixSorter.java:209)
at org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:148)
at org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:155)
at org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:128)
at org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:121)
at org.apache.lucene.util.bkd.MutablePointsReaderUtils.sort(MutablePointsReaderUtils.java:90)
at org.apache.lucene.util.bkd.BKDWriter.writeField1Dim(BKDWriter.java:497)
at org.apache.lucene.util.bkd.BKDWriter.writeField(BKDWriter.java:427)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:105)
at org.apache.lucene.index.PointValuesWriter.flush(PointValuesWriter.java:183)
at org.apache.lucene.index.DefaultIndexingChain.writePoints(DefaultIndexingChain.java:206)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:141)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:470)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
at org.apache.lucene.index.DocumentsWriter.flushOneDWPT(DocumentsWriter.java:257)
at org.apache.lucene.index.IndexWriter.flushNextBuffer(IndexWriter.java:3157)
at com.amazon.lucene.index.Indexer.lambda$commit$0(Indexer.java:1129)
```
The exception makes no sense to me: when I look at `MSBRadixSorter.computeCommonPrefixLengthAndBuildHistogram` at that line it does NOT invoke `Sorter.heapify` so I'm mystified. Maybe this is a JVM bug ...
---
Migrated from [LUCENE-8822](https://issues.apache.org/jira/browse/LUCENE-8822) by Michael McCandless (@mikemccand)
Contributor guide
Research direction
Start with the stack trace in MSBRadixSorter.java, Sorter.java, MutablePointsReaderUtils.java, and BKDWriter.java, then compare the reported Lucene 7.x snapshot and OpenJDK 11 context. Reproduce the IndexWriter flush failure if possible and determine whether the trace indicates a Lucene defect or a JVM issue; done means the cause and next action are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100