apache / apache/lucene

Incorrect behavior for TestLaoBreakIterator.isWord() [LUCENE-5076]

Open
#6,140 1 comment 0 reactions 0 assignees View on GitHub
affects-version:4.3.1 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The incorrect behavior appears in version 4.3.1 and in revision
1496055.

Method "TestLaoBreakIterator.isWord" contains this loop:

```java
for (int i = start; i < end; i += UTF16.getCharCount(codepoint)) {
codepoint = UTF16.charAt(text, 0, end, start);

if (UCharacter.isLetterOrDigit(codepoint))
return true;
}
```

It appears that the code is reading only one character again and
again, irrespective of "i". This looks incorrect. I think the code
inside the loop should use "i", e.g., read characters based on "i".

If the intended behavior is to read only one character, then the loop
should not be necessary.

A similar problem appears in method
"BreakIteratorWrapper.BIWrapper.calcStatus" for this loop:

```java
for (int i = begin; i < end; i += UTF16.getCharCount(codepoint)) {
codepoint = UTF16.charAt(text, 0, end, begin);

if (UCharacter.isDigit(codepoint))
return RuleBasedBreakIterator.WORD_NUMBER;
else if (UCharacter.isLetter(codepoint)) {
// TODO: try to separately specify ideographic, kana?
// [currently all bundled as letter for this case]
return RuleBasedBreakIterator.WORD_LETTER;
}
}
```

Again, the computation inside the loop does not use "i", which seems
incorrect. It appears that the code is reading only one character
again and again, irrespective of "i".

---
Migrated from [LUCENE-5076](https://issues.apache.org/jira/browse/LUCENE-5076) by Adrian Nistor
Environment:
```
any
```

Contributor guide

Open the contributing guide

Research direction

Start by inspecting TestLaoBreakIterator.isWord and BreakIteratorWrapper.BIWrapper.calcStatus, focusing on the loop bodies and their character-position arguments. Run the existing break-iterator tests, then add or update coverage for Lao word detection and word-status calculation so the corrected behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.