apache / apache/lucene

getOffsetGap should not be called for non-anaylyzed fields [LUCENE-2801]

Open
#3,875 0 comments 0 reactions 0 assignees View on GitHub
affects-version:3.0.3 legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

from: #3311

Since Lucene 3.0.3, when a PerFieldAnalyzerWrapper is constructed with a null defaultAnalyzer it will NPE when DocInverterPerField calls:

```Java
fieldState.offset += docState.analyzer.getOffsetGap(field);
```

This block should first check that the field is analyzed, or the javadoc on PerFieldAnalyzerWrapper could mention that a null defaultAnalyzer is disallowed.

Also, the main reason for checking for isAnalyzed, from Uwe Schindler in #3311
> One problem coming from not checking for "analyzed" is this:
> You add a field indexed and it gets analyzed by PFAW - After that you add the same field name stored-only (which is perfectly legal and often used, e\.g\. when the stored value is binary or in some other format and does not correspond to the indexed text), the positionIncrement is increased\. After that you again add another instance of the same field as indexed-only, which also increases posIncr\. So you have 2 times the gap between both indexed sub-fields\. This is definitely wrong\.

---
Migrated from [LUCENE-2801](https://issues.apache.org/jira/browse/LUCENE-2801) by Nick Pellow
Linked issues:
- #3311

Contributor guide

Open the contributing guide

Research direction

Start in DocInverterPerField at the getOffsetGap call and read how PerFieldAnalyzerWrapper handles a null defaultAnalyzer and non-analyzed fields. Compare the behavior described in linked issue #3311; done means avoiding the null-analyzer failure and preventing offset or position gaps for non-analyzed field instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.