apache / apache/lucene

Enable extraction of originating term for ICU collation keys [LUCENE-4799]

Open
#5,864 5 comments 0 reactions 0 assignees View on GitHub
affects-version:4.1 legacy-jira-label:collator legacy-jira-label:facet legacy-jira-priority:Minor module:core/other type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

By concatenating generated ICU collation keys bytes with the originating term, it is possible to extract the originating term at a later time. This makes it possible to build a collator sorted facet field and similar multi-value/document structures.

ICU collation keys are guaranteed to be terminated by a 0 (https://ssl.icu-project.org/apiref/icu4j48rc1/com/ibm/icu/text/CollationKey.html) and since comparison of keys stop when a 0 is encountered, the addition of the originating term does not affect sort order. As 0 are _only_ used for termination in the key bytes, the extraction of the originating term is unambiguous.

---
Migrated from [LUCENE-4799](https://issues.apache.org/jira/browse/LUCENE-4799) by Toke Eskildsen (@tokee), 1 vote
Attachments: [LUCENE-4799.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4799/LUCENE-4799.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ICU CollationKey behavior described in the issue and the attached LUCENE-4799.patch. Trace how Lucene generates and compares ICU collation key bytes, then determine where the originating term should be appended and extracted. Done means the term can be recovered unambiguously without changing collation sort order.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.