apache / apache/lucene

TestCollationDocValuesField.testRanges failure [LUCENE-6206]

Open
#7,268 10 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This test failed in branch_5x:

```
ant test -Dtestcase=TestCollationDocValuesField -Dtests.method=testRanges -Dtests.seed=8729A11DAE9AE179 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt -Dtests.timezone=Etc/GMT-8 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```

The problem is, with lithuanian collator, the generated collationkey does not agree with collation order.

In other words: collator.compare("yi", "iy") != collator.getCollationKey("yi").compareTo(collator.getCollationKey("iy"))

Simple test:

```Java
import java.text.Collator;
import java.util.Locale;

public class TestLithuanian {
public static void main(String args[]) throws Exception {
Collator collator = Collator.getInstance(new Locale("lt"));
int value = collator.compare("yi", "iy");
System.out.println(value); // returns 1
int value2 = collator.getCollationKey("yi").compareTo(collator.getCollationKey("iy"));
System.out.println(value2); // returns -1
}
}
```

Note that if you change java.text.Collator to com.ibm.icu.text.Collator, they are consistent. Looks like a bug in JDK collation.

---
Migrated from [LUCENE-6206](https://issues.apache.org/jira/browse/LUCENE-6206) by Robert Muir (@rmuir), updated Aug 31 2015

Contributor guide

Open the contributing guide

Research direction

Start by running the reported ant command for TestCollationDocValuesField.testRanges with the Lithuanian locale and seed. Read the test and compare java.text.Collator ordering with its CollationKey ordering, then check the noted com.ibm.icu.text.Collator behavior. Done requires a confirmed explanation or a project change that prevents this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.