apache / apache/lucene

CombiningFilter to recombine tokens into a single token for sorting [LUCENE-3413]

Open
#4,486 21 comments 0 reactions 0 assignees View on GitHub
affects-version:2.9.3 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I whipped up this CombiningFilter for the following use case:

I've got a bunch of titles of e.g., Books, such as:

The Grapes of Wrath
Tommy Tommerson saves the World
Top of the World
The Tales of Beedle the Bard
Born Free

etc.

I want to sort these titles using a String field that includes stopword analysis (e.g., to remove "The"), and synonym filtering (e.g., for grouping), etc. I created an analysis chain in Solr for this that was based off of **alphaOnlySort**, which looks like this:

```xml












```

The issue with alphaOnlySort is that it doesn't support stopword remove or synonyms because those are based on the original token level instead of the full strings produced by the KeywordTokenizer (which does not do tokenization). I needed a filter that would allow me to change alphaOnlySort and its analysis chain from using KeywordTokenizer to using WhitespaceTokenizer, and then a way to recombine the tokens at the end. So, take "The Grapes of Wrath". I needed a way for it to get turned into:

```
grapes of wrath
```

And then to combine those tokens into a single token:

```
grapesofwrath
```

The attached CombiningFilter takes care of that. It doesn't do it super efficiently I'm guessing (since I used a StringBuffer), but I'm open to suggestions on how to make it better.

One other thing is that apparently this analyzer works fine for analysis (e.g., it produces the desired tokens), however, for sorting in Solr I'm getting null sort tokens. Need to figure out why.

Here ya go!

---
Migrated from [LUCENE-3413](https://issues.apache.org/jira/browse/LUCENE-3413) by Chris A. Mattmann, updated Jan 09 2013
Attachments: [LUCENE-3413.Mattmann.090311.patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3413/LUCENE-3413.Mattmann.090311.patch.txt), [LUCENE-3413.Mattmann.090511.patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3413/LUCENE-3413.Mattmann.090511.patch.txt)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the two attached LUCENE-3413 patch files and the alphaOnlySort analysis chain described in the issue. Reproduce the analyzer with stopword and synonym filtering, then investigate why Solr produces null sort tokens. Done means tokens are recombined into one sortable token with the intended normalized value.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.