[FEA] [Java] Threadpool for creating indexes?
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
For setting the concurrency during building, we use omp_set_num_threads. However, one drawback with this approach could be when multiple indexes are being build in parallel (which could be a realistic use-case in Solr etc.), there would end up being n * omp_set_num_threads where n is the number of parallel indexes being built. This value could easily exceed the number of available CPUs and may cause starvation for the application. One way could be to use a thread pool shared among the parallel indexing processes so that the maximum number of threads can be restricted to the threadpool size.
I'm putting my thoughts on this out for discussion. Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; start by locating the current omp_set_num_threads usage and the parallel index-building entry points. Review how concurrent index builds are initiated and document the scope and acceptance criteria for a shared thread-pool design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100