CombiningFilter to recombine tokens into a single token for sorting [LUCENE-3413]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I whipped up this CombiningFilter for the following use case:
I've got a bunch of titles of e.g., Books, such as:
The Grapes of Wrath
Tommy Tommerson saves the World
Top of the World
The Tales of Beedle the Bard
Born Free
etc.
I want to sort these titles using a String field that includes stopword analysis (e.g., to remove "The"), and synonym filtering (e.g., for grouping), etc. I created an analysis chain in Solr for this that was based off of **alphaOnlySort**, which looks like this:
```xml
```
The issue with alphaOnlySort is that it doesn't support stopword remove or synonyms because those are based on the original token level instead of the full strings produced by the KeywordTokenizer (which does not do tokenization). I needed a filter that would allow me to change alphaOnlySort and its analysis chain from using KeywordTokenizer to using WhitespaceTokenizer, and then a way to recombine the tokens at the end. So, take "The Grapes of Wrath". I needed a way for it to get turned into:
```
grapes of wrath
```
And then to combine those tokens into a single token:
```
grapesofwrath
```
The attached CombiningFilter takes care of that. It doesn't do it super efficiently I'm guessing (since I used a StringBuffer), but I'm open to suggestions on how to make it better.
One other thing is that apparently this analyzer works fine for analysis (e.g., it produces the desired tokens), however, for sorting in Solr I'm getting null sort tokens. Need to figure out why.
Here ya go!
---
Migrated from [LUCENE-3413](https://issues.apache.org/jira/browse/LUCENE-3413) by Chris A. Mattmann, updated Jan 09 2013
Attachments: [LUCENE-3413.Mattmann.090311.patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3413/LUCENE-3413.Mattmann.090311.patch.txt), [LUCENE-3413.Mattmann.090511.patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3413/LUCENE-3413.Mattmann.090511.patch.txt)
Contributor guide
Research direction
Start by reviewing the two attached LUCENE-3413 patch files and the alphaOnlySort analysis chain described in the issue. Reproduce the analyzer with stopword and synonym filtering, then investigate why Solr produces null sort tokens. Done means tokens are recombined into one sortable token with the intended normalized value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100