apache / apache/lucene

Should FilteringTokenFilter handle positionLength [LUCENE-8250]

Open
#9,297 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

FilteringTokenFilter does not handle the position length graph attribute when removing a token from the stream. This doesn't work well with graph token stream that sets position length since removing a token from the stream can invalidate the position length set on the previous tokens.
This issue was first discussed in #5137 but it has a different purpose which is why I am opening a new issue here.

---
Migrated from [LUCENE-8250](https://issues.apache.org/jira/browse/LUCENE-8250) by Jim Ferenczi (@jimczi), updated Apr 13 2018
Attachments: [LUCENE-8250.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8250/LUCENE-8250.patch)

Contributor guide

Open the contributing guide

Research direction

Start by locating FilteringTokenFilter and read how it removes tokens while position lengths are preserved. Review the issue's graph-token-stream behavior and the attached LUCENE-8250.patch; done means removed tokens no longer invalidate position lengths on preceding tokens.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.