MemoryIndex and FVH don't play along with multi-value fields [LUCENE-5011]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 86
Description
When multi-value fields are indexed to a MemoryIndex, positions are computed correctly on search but the start and end offsets and the values array index aren't correct.
Comparing the same execution path for IndexReader on a Directory impl and MemoryIndex (same document, same query, same analyzer, different Index impl), the difference first shows in FieldTermStack.java line 125:
termList.add( new TermInfo( term, dpEnum.startOffset(), dpEnum.endOffset(), pos, weight ) );
dpEnum.startOffset() and dpEnum.endOffset don't match between implementations.
This looks like a bug in MemoryIndex, which doesn't seem to handle tokenized multi-value fields all too well when positions and offsets are required.
I should also mention we are using an Analyzer which outputs several tokens at a position (a la SynonymFilter), but I don't believe this is related.
---
Migrated from [LUCENE-5011](https://issues.apache.org/jira/browse/LUCENE-5011) by Itamar Syn-Hershko, updated Jul 31 2014
Contributor guide
Research direction
Start at FieldTermStack.java line 125 and compare the same document, query, analyzer, and token stream through Directory and MemoryIndex. Inspect MemoryIndex handling of tokenized multi-value fields, positions, offsets, and values-array indexes; done means the implementations produce matching offsets and value indexes while preserving the currently correct positions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100