apache / apache/lucene

Make WordDelimiterGraphFilter a Tokenizer [LUCENE-8516]

Open
#9,562 9 comments 0 reactions 1 assignee Claimed by @romseygeek View on GitHub
legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Being able to split tokens up at arbitrary points in a filter chain, in effect adding a second round of tokenization, can cause any number of problems when trying to keep tokenstreams to contract. The most common offender here is the WordDelimiterGraphFilter, which can produce broken offsets in a wide range of situations.

We should make WDGF a Tokenizer in its own right, which should preserve all the functionality we need, but make reasoning about the resulting tokenstream much simpler.

---
Migrated from [LUCENE-8516](https://issues.apache.org/jira/browse/LUCENE-8516) by Alan Woodward (@romseygeek), updated Oct 04 2018
Attachments: [LUCENE-8516.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8516/LUCENE-8516.patch)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.