apache / apache/lucene

UnsupportedOperationException: unused: not a comparsion-based sort during IndexWriter flush [LUCENE-8822]

Open
#9,866 0 comments 0 reactions 0 assignees View on GitHub
affects-version:7.6 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We hit this very strange exception in production 7.x snapshot (near 7.6), OpenJDK 11:
```
Caused by: java.lang.UnsupportedOperationException: unused: not a comparison-based sort
at org.apache.lucene.util.MSBRadixSorter.compare(MSBRadixSorter.java:115)
at org.apache.lucene.util.Sorter.siftDown(Sorter.java:235)
at org.apache.lucene.util.Sorter.heapify(Sorter.java:228)
at org.apache.lucene.util.MSBRadixSorter.computeCommonPrefixLengthAndBuildHistogram(MSBRadixSorter.java:209)
at org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:148)
at org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:155)
at org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:128)
at org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:121)
at org.apache.lucene.util.bkd.MutablePointsReaderUtils.sort(MutablePointsReaderUtils.java:90)
at org.apache.lucene.util.bkd.BKDWriter.writeField1Dim(BKDWriter.java:497)
at org.apache.lucene.util.bkd.BKDWriter.writeField(BKDWriter.java:427)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:105)
at org.apache.lucene.index.PointValuesWriter.flush(PointValuesWriter.java:183)
at org.apache.lucene.index.DefaultIndexingChain.writePoints(DefaultIndexingChain.java:206)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:141)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:470)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
at org.apache.lucene.index.DocumentsWriter.flushOneDWPT(DocumentsWriter.java:257)
at org.apache.lucene.index.IndexWriter.flushNextBuffer(IndexWriter.java:3157)
at com.amazon.lucene.index.Indexer.lambda$commit$0(Indexer.java:1129)
```
The exception makes no sense to me: when I look at `MSBRadixSorter.computeCommonPrefixLengthAndBuildHistogram` at that line it does NOT invoke `Sorter.heapify` so I'm mystified.  Maybe this is a JVM bug ...

---
Migrated from [LUCENE-8822](https://issues.apache.org/jira/browse/LUCENE-8822) by Michael McCandless (@mikemccand)

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace in MSBRadixSorter.java, Sorter.java, MutablePointsReaderUtils.java, and BKDWriter.java, then compare the reported Lucene 7.x snapshot and OpenJDK 11 context. Reproduce the IndexWriter flush failure if possible and determine whether the trace indicates a Lucene defect or a JVM issue; done means the cause and next action are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.