apache / apache/lucene

NPE in o.a.l.codecs.perfield.PerFieldPostingsFormat [LUCENE-8666]

Open
#9,712 1 comment 0 reactions 0 assignees View on GitHub
affects-version:7.5 affects-version:9.0 legacy-jira-label:diffblue legacy-jira-label:newdev legacy-jira-label:patch-available legacy-jira-priority:Minor module:core/codecs type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Requesting this URL in SOLR gives a 500 error with a stack trace pointing to Lucene:

`http://localhost:8983/solr/films/select?q=\{!complexphrase}genre:"-om*"`

The stack trace is (cut down to the reasonably relevant part):

{{java.lang.NullPointerException\n\tat java.util.TreeMap.getEntry(TreeMap.java:347)
at java.util.TreeMap.get(TreeMap.java:278)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:311)
at org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)
at org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:351)
at org.apache.lucene.index.ExitableDirectoryReader$ExitableFilterAtomicReader.terms(ExitableDirectoryReader.java:91)
at org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight.getSpans(SpanNearQuery.java:208)
at org.apache.lucene.search.spans.SpanNotQuery$SpanNotWeight.getSpans(SpanNotQuery.java:127)
at org.apache.lucene.search.spans.SpanWeight.scorer(SpanWeight.java:135)
at org.apache.lucene.search.spans.SpanWeight.scorer(SpanWeight.java:46)
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:649)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443)
at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:200)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1604)}}`The error is actually a bit deeper and can be traced back to the o.a.l.queryparser.complexPhrase.ComplexPhraseQueryParser class.`

Handling this query involves constructing a SpanQuery, which happens in the rewrite method of ComplexPhraseQueryParser. In particular, the expression is decomposed into a BooleanQuery, which has exactly one clause, namely the negative clause -genre:”om\*”. The rewrite method then further transforms this into a SpanQuery; in this case, it goes into the path that handles complex queries with both positive and negative clauses. It extracts the subset of positive clauses - note that this set of clauses is empty for this query. The positive clauses are then combined into a SpanNearQuery (around line 340), which is then used to build a SpanNotQuery. Further down the line, the field attribute of the SpanNearQuery is accessed and used as an index into a TreeMap. But since we had an empty set of positive clauses, the SpanNearQuery does not have its field attribute set, so we get a null here - this leads to an exception. A possible fix would be to detect the situation where we have an empty set of positive clauses and include a single synthetic clause that matches either everything or nothing. See attached file 0001-Fix-NullPointerException.patch.

This bug was found using [Diffblue Microservices Testing](http://www.diffblue.com/labs). Find more information on this [test campaign](https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results).

---
Migrated from [LUCENE-8666](https://issues.apache.org/jira/browse/LUCENE-8666) by Johannes Kloos, 1 vote, updated Apr 01 2019
Environment:
```
Running on Unix, using a git checkout close to master.

h2. Steps to reproduce
* Build commit ea2c8ba of Solr as described in the section below.
* Build the films collection as described below.
* Start the server using the command {{“./bin/solr start -f -p 8983 -s /tmp/home”}}
* Request the URL above.

h2. Compiling the server

{noformat}
git clone https://github.com/apache/lucene-solr
cd lucene-solr
git checkout ea2c8ba
ant compile
cd solr
ant server
{noformat}

h2. Building the collection

We followed Exercise 2 from the quick start tutorial ([http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2]) - for reference, I have attached a copy of the database.

{noformat}
mkdir -p /tmp/home
echo '' > /tmp/home/solr.xml
{noformat}

In one terminal start a Solr instance in foreground:

{noformat}
./bin/solr start -f -p 8983 -s /tmp/home
{noformat}

In another terminal, create a collection of movies, with no shards and no replication:

{noformat}
bin/solr create -c films
curl -X POST -H 'Content-type:application/json' --data-binary '\{"add-field": {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}' http://localhost:8983/solr/films/schema
curl -X POST -H 'Content-type:application/json' --data-binary '{"add-copy-field" : {"source":"*","dest":"_text_"}}' [http://localhost:8983/solr/films/schema]
./bin/post -c films example/films/films.json
{noformat}
```

Attachments: [0001-Fix-NullPointerException.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8666/0001-Fix-NullPointerException.patch), [home.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8666/home.zip)

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.lucene.queryparser.complexPhrase.ComplexPhraseQueryParser, especially rewrite and the path around line 340 that handles negative-only clauses. Reproduce the Solr request against the provided setup and inspect how the resulting SpanNearQuery reaches PerFieldPostingsFormat.FieldsReader. Done means the query no longer returns a 500 error or NullPointerException; the attached patch is available for comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.