apache / apache/lucene

CloseableThreadLocal maybePurge could be too expensive [LUCENE-4474]

Open
#5,540 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Was doing some tests with geonames database (19 fields, just using StandardAnalyzer), and noticed this in the profiler.

It could be a ghost, but we should investigate anyway.

It seems ridiculous for a situation like mine:
- indexing with one thread
- every 40 Analyzer.tokenStream() calls [basically every other doc], this thing is called
- it gets iterators over the map, checks threads, this and that. but of course there is only one thread!

Maybe its a good idea if it checks size() first or something. at least dont do this stuff if size() == 1, as I bet a lot of people index with a single thread.

Or maybe all this stuff is really cheap and its just a ghost.

---
Migrated from [LUCENE-4474](https://issues.apache.org/jira/browse/LUCENE-4474) by Robert Muir (@rmuir)

Contributor guide

Open the contributing guide

Research direction

Start at CloseableThreadLocal.maybePurge and trace its calls from Analyzer.tokenStream, using the reported single-thread indexing scenario as the benchmark case. Profile or inspect the purge path to determine whether iterator and thread checks are measurably expensive; done means a confirmed diagnosis and a clearly justified change or finding.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.