Improve reuse of StringTokenStream [LUCENE-10203]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This issue is a follow-up to https://lists.apache.org/thread.html/rdcc6bd085a0e8ac6db22a1ef7dd3228197481b62bec3c6fe4972e50a%40%3Cdev.lucene.apache.org%3E.
StringField has a different mechanism for reusing token streams compared to TextField: while TextField relies on `Analyzer#reuseStrategy` to reuse token streams across inputs, StringField relies on IndexingChain passing the previously produced token stream as the `reuse` parameter of `IndexableField#tokenStream`. However one downside of this approach is that it can only reuse token streams within a single segment. And some nightly profiles suggest that not reusing across segments still gives room for attribute initialization to be a hotspot.
---
Migrated from [LUCENE-10203](https://issues.apache.org/jira/browse/LUCENE-10203) by Adrien Grand (@jpountz)
Contributor guide
Research direction
Start by reading StringField and TextField, then follow Analyzer#reuseStrategy, IndexingChain, and IndexableField#tokenStream to understand the two reuse paths. The linked mailing-list discussion is the only stated design context. Done would mean enabling reuse across segments and reducing attribute-initialization overhead, but this issue does not define a concrete implementation or test target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100