make CharTokenizer.MAX_WORD_LEN parametrizable [LUCENE-2407]
Open
affects-version:3.0.1
legacy-jira-fix-version:4.9
legacy-jira-fix-version:6.0
legacy-jira-label:dead
legacy-jira-priority:Minor
module:analysis
type:enhancement
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
as discussed here http://n3.nabble.com/are-long-words-split-into-up-to-256-long-tokens-tp739914p739914.html it would be nice to be able to parametrize that value.
---
Migrated from [LUCENE-2407](https://issues.apache.org/jira/browse/LUCENE-2407) by jmwap, updated May 09 2016
Contributor guide
Research direction
Start by locating CharTokenizer and its MAX_WORD_LEN value, then read the linked discussion for the expected behavior around long words. Done means the limit can be configured rather than remaining fixed, while long-word tokenization behavior is verified against the requested setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100