apache / apache/lucene

Mishandling of Integer.MIN_VALUE in FuzzySet leads to AssertionError [LUCENE-6788]

Open
#7,846 2 comments 0 reactions 0 assignees View on GitHub
affects-version:4.10.4 affects-version:6.0 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Reindexing some data in the DataStax Enterprise Search product (which uses Solr) led to these stack traces:

ERROR [Lucene Merge Thread #13430] 2015-09-08 11:14:36,582 CassandraDaemon.java (line 258) Exception in thread Thread[Lucene Merge Thread #13430,6,main]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.AssertionError
at org.apache.lucene.codecs.bloom.FuzzySet.mayContainValue(FuzzySet.java:216)
at org.apache.lucene.codecs.bloom.FuzzySet.contains(FuzzySet.java:165)
at org.apache.lucene.codecs.bloom.BloomFilteringPostingsFormat$BloomFilteredFieldsProducer$BloomFilteredTermsEnum.seekExact(BloomFilteringPostingsFormat.java:351)
at org.apache.lucene.index.BufferedUpdatesStream.applyTermDeletes(BufferedUpdatesStream.java:414)
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:283)
at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3838)
at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3799)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3651)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

In tracking down the cause of the stack trace, I noticed this:
https://github.com/apache/lucene-solr/blob/trunk/lucene/codecs/src/java/org/apache/lucene/codecs/bloom/FuzzySet.java#L164

It is possible for the Murmur2 hash to return Integer.MIN_VALUE (e.g. when hashing "WeH44wlbCK"). Multiplying Integer.MIN_VALUE by -1 returns Integer.MIN_VALUE again, so the "positiveHash >= 0" assertion at line 217 fails.

We could special-case Integer.MIN_VALUE, map it to 42 or some other magic number... since the same "\* -1" logic appears on line 236 perhaps it should be part of the hash function?

---
Migrated from [LUCENE-6788](https://issues.apache.org/jira/browse/LUCENE-6788) by Robert Tarrall (@tarrall), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start in lucene/codecs/src/java/org/apache/lucene/codecs/bloom/FuzzySet.java, especially the hash handling around lines 164, 217, and 236. Reproduce the Integer.MIN_VALUE case with the reported “WeH44wlbCK” input and inspect existing FuzzySet tests or callers. Done means the value no longer triggers the assertion during lookup or merge processing, with regression coverage in a suitable test location.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.