apache / apache/lucene

Exceptions handling in methods of SegmentingTokenizerBase [LUCENE-9588]

Open
#10,628 4 comments 0 reactions 0 assignees View on GitHub
affects-version:8.6.3 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The current interface of **setNextSentence** and **incrementWord** methods in **SegmentingTokenizerBase** do not define the checked exceptions, which makes it troublesome to be inherited.

For example, if we override the incrementWord  with a logic that invoke  incrementToken on another tokenizer, the incrementToken raises the IOException but the incrementWord is not defined to handle it.

I think having setNextSentence and incrementWord handle the IOException would make the SegmentingTokenizerBase easier to be used.

---
Migrated from [LUCENE-9588](https://issues.apache.org/jira/browse/LUCENE-9588) by Nguyen Minh Gia Huy, updated Nov 05 2020

Contributor guide

Open the contributing guide

Research direction

Start at SegmentingTokenizerBase and inspect the setNextSentence and incrementWord method signatures, then trace the incrementToken call mentioned in the issue. The work is done when these methods can handle the IOException needed by inherited implementations without breaking the tokenizer API; verify the affected Java tests or compilation afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.