Deprecate Token class and remove in trunk [LUCENE-3844]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
After issues like #4916, introducing new attributes, we should remove Token class in trunk, as it leads to code that ignores those new attributes (like PositionLengthAttribute, ScriptAttribute, KeywordAttribute,...). If you want a holder for all Attributes a TokenStream, use TS.cloneAttributes().
---
Migrated from [LUCENE-3844](https://issues.apache.org/jira/browse/LUCENE-3844) by Uwe Schindler (@uschindler), updated May 09 2016
Linked issues:
- #4916
Contributor guide
Research direction
Start by locating the Token class and tracing its usages in the Lucene source, especially TokenStream-related code. Check how TS.cloneAttributes() is used as the replacement, then verify that Token is removed without leaving code that ignores PositionLengthAttribute, ScriptAttribute, or KeywordAttribute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100