Can we take advantage of the Vector API for text analysis?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
`StandardTokenizer` is likely our most widely used tokenizer, and is reported as the main bottleneck for indexing in our nightly benchmarks, see e.g. top 5 CPU users for the 1kB Wikipedia corpus on yesterday's run:
```
PERCENT CPU SAMPLES STACK
10.54% 51017 org.apache.lucene.analysis.standard.StandardTokenizerImpl#getNextToken()
6.99% 33852 org.apache.lucene.index.IndexingChain$PerField#invertTokenStream()
6.47% 31309 org.apache.lucene.index.TermsHashPerField#writeByte()
5.00% 24183 org.apache.lucene.util.BytesRefHash#equals()
4.38% 21215 java.lang.Character#codePointAtImpl()
```
Intuitively, this kind of workload is amenable to vectorization, could we take advantage of vectorization to speed up text analysis and thus indexing?
Contributor guide
Research direction
Start with StandardTokenizer and StandardTokenizerImpl#getNextToken, then inspect the nightly benchmark data for the 1kB Wikipedia corpus to understand where vectorization might apply. Done would require demonstrating a meaningful text-analysis or indexing speedup without changing tokenizer behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100