Randomize indexed collation key testing [LUCENE-2798]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Robert Muir noted on #lucene IRC channel today that Lucene's indexed collation key testing is currently fragile (for example, they had to be revisited when Robert upgraded the ICU dependency in #3871 because of Unicode 6.0 collation changes) and coverage is trivial (only 5 locales tested, and no collator options are exercised). This affects both the JDK implementation in `modules/analysis/common/` and the ICU implementation under `modules/icu/`.
The key thing to test is that the order of the indexed terms is the same as that provided by the Collator itself. Instead of the current set of static tests, this could be achieved via indexing randomly generated terms' collation keys (and collator options) and then comparing the index terms' order to the order provided by the Collator over the original terms.
Since different terms may produce the same collation key, however, the order of indexed terms is inherently unstable. When performing runtime collation, the Collator addresses the sort stability issue by adding a secondary sort over the normalized original terms. In order to directly compare Collator's sort with Lucene's collation key sort, a secondary sort will need to be applied to Lucene's indexed terms as well. Robert has suggested indexing the original terms in addition to their collation keys, then using a Sort over the original terms as the secondary sort.
Another complication: Lucene 3.X uses Java's UTF-16 term comparison, and trunk uses UTF-8 order, so the implemented secondary sort will need to respect that.
From #lucene:
> rmuir__: so i think we have to on 3\.x, sort the 'expected list' with Collator\.compare, if thats equal, then as a tiebreak use String\.compareTo
> rmuir__: and in the index sort on the collated field, followed by the original term
> rmuir__: in 4\.x we do the same thing, but dont use String\.compareTo as the tiebreak for the expected list
> rmuir__: instead compare codepoints (iterating character\.codepointAt, or comparing \.getBytes("UTF-8"))
---
Migrated from [LUCENE-2798](https://issues.apache.org/jira/browse/LUCENE-2798) by Steven Rowe (@sarowe), updated May 09 2016
Attachments: [LUCENE-2798.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2798/LUCENE-2798.patch) (versions: 2)
Linked issues:
- #3871
Contributor guide
Assessment
This issue has not been assessed yet.