apache / apache/lucene

MemoryIndex and FVH don't play along with multi-value fields [LUCENE-5011]

Open
#6,075 3 comments 0 reactions 0 assignees View on GitHub
affects-version:4.3 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 12h
Merged PRs (30d)
86

Description

When multi-value fields are indexed to a MemoryIndex, positions are computed correctly on search but the start and end offsets and the values array index aren't correct.

Comparing the same execution path for IndexReader on a Directory impl and MemoryIndex (same document, same query, same analyzer, different Index impl), the difference first shows in FieldTermStack.java line 125:

termList.add( new TermInfo( term, dpEnum.startOffset(), dpEnum.endOffset(), pos, weight ) );

dpEnum.startOffset() and dpEnum.endOffset don't match between implementations.

This looks like a bug in MemoryIndex, which doesn't seem to handle tokenized multi-value fields all too well when positions and offsets are required.

I should also mention we are using an Analyzer which outputs several tokens at a position (a la SynonymFilter), but I don't believe this is related.

---
Migrated from [LUCENE-5011](https://issues.apache.org/jira/browse/LUCENE-5011) by Itamar Syn-Hershko, updated Jul 31 2014

Contributor guide

Open the contributing guide

Research direction

Start at FieldTermStack.java line 125 and compare the same document, query, analyzer, and token stream through Directory and MemoryIndex. Inspect MemoryIndex handling of tokenized multi-value fields, positions, offsets, and values-array indexes; done means the implementations produce matching offsets and value indexes while preserving the currently correct positions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.