apache / apache/lucene

Enhance WordDelimiterFilter to skip operation on tokens marked as keywords. [LUCENE-6903]

Open
#7,961 2 comments 0 reactions 0 assignees View on GitHub
affects-version:5.3.1 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We have a github pull request for a feature to allow WDF to skip operation when the keyword marker filter has marked tokens.

https://github.com/apache/lucene-solr/pull/210

This will also add a new configuration parameter called "splitKeywordTokens" that will default to false in 6.0 and true in earlier versions.

---
Migrated from [LUCENE-6903](https://issues.apache.org/jira/browse/LUCENE-6903) by Shawn Heisey (@elyograg)
Attachments: [LUCENE-6903.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6903/LUCENE-6903.patch)

Contributor guide

Open the contributing guide

Research direction

Review pull request 210 and the attached LUCENE-6903.patch first, since the proposed implementation is already linked from the issue. Confirm that WordDelimiterFilter skips marked keyword tokens and that the splitKeywordTokens parameter has the stated defaults for Lucene 6.0 and earlier versions; the existing pull request should show the relevant tests and files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.