apache / apache/lucene

Improve reuse of StringTokenStream [LUCENE-10203]

Open
#11,240 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This issue is a follow-up to https://lists.apache.org/thread.html/rdcc6bd085a0e8ac6db22a1ef7dd3228197481b62bec3c6fe4972e50a%40%3Cdev.lucene.apache.org%3E.

StringField has a different mechanism for reusing token streams compared to TextField: while TextField relies on `Analyzer#reuseStrategy` to reuse token streams across inputs, StringField relies on IndexingChain passing the previously produced token stream as the `reuse` parameter of `IndexableField#tokenStream`. However one downside of this approach is that it can only reuse token streams within a single segment. And some nightly profiles suggest that not reusing across segments still gives room for attribute initialization to be a hotspot.

---
Migrated from [LUCENE-10203](https://issues.apache.org/jira/browse/LUCENE-10203) by Adrien Grand (@jpountz)

Contributor guide

Open the contributing guide

Research direction

Start by reading StringField and TextField, then follow Analyzer#reuseStrategy, IndexingChain, and IndexableField#tokenStream to understand the two reuse paths. The linked mailing-list discussion is the only stated design context. Done would mean enabling reuse across segments and reducing attribute-initialization overhead, but this issue does not define a concrete implementation or test target.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.