CloseableThreadLocal maybePurge could be too expensive [LUCENE-4474]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Was doing some tests with geonames database (19 fields, just using StandardAnalyzer), and noticed this in the profiler.
It could be a ghost, but we should investigate anyway.
It seems ridiculous for a situation like mine:
- indexing with one thread
- every 40 Analyzer.tokenStream() calls [basically every other doc], this thing is called
- it gets iterators over the map, checks threads, this and that. but of course there is only one thread!
Maybe its a good idea if it checks size() first or something. at least dont do this stuff if size() == 1, as I bet a lot of people index with a single thread.
Or maybe all this stuff is really cheap and its just a ghost.
---
Migrated from [LUCENE-4474](https://issues.apache.org/jira/browse/LUCENE-4474) by Robert Muir (@rmuir)
Contributor guide
Research direction
Start at CloseableThreadLocal.maybePurge and trace its calls from Analyzer.tokenStream, using the reported single-thread indexing scenario as the benchmark case. Profile or inspect the purge path to determine whether iterator and thread checks are measurably expensive; done means a confirmed diagnosis and a clearly justified change or finding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100