apache / apache/lucene

Can we take advantage of the Vector API for text analysis?

Open
#12,393 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

`StandardTokenizer` is likely our most widely used tokenizer, and is reported as the main bottleneck for indexing in our nightly benchmarks, see e.g. top 5 CPU users for the 1kB Wikipedia corpus on yesterday's run:

```
PERCENT CPU SAMPLES STACK
10.54% 51017 org.apache.lucene.analysis.standard.StandardTokenizerImpl#getNextToken()
6.99% 33852 org.apache.lucene.index.IndexingChain$PerField#invertTokenStream()
6.47% 31309 org.apache.lucene.index.TermsHashPerField#writeByte()
5.00% 24183 org.apache.lucene.util.BytesRefHash#equals()
4.38% 21215 java.lang.Character#codePointAtImpl()
```

Intuitively, this kind of workload is amenable to vectorization, could we take advantage of vectorization to speed up text analysis and thus indexing?

Contributor guide

Open the contributing guide

Research direction

Start with StandardTokenizer and StandardTokenizerImpl#getNextToken, then inspect the nightly benchmark data for the 1kB Wikipedia corpus to understand where vectorization might apply. Done would require demonstrating a meaningful text-analysis or indexing speedup without changing tokenizer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.