Hook for merge thread pool
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
In opensearch k-NN, we want merge operations to be executed with a dynamic number of threads. To do this, we are giving a TaskExecutor a dynamic number of threads here: https://github.com/opensearch-project/k-NN/blob/e327675aa7c404ab0cac533a826bca0febb53fdd/src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/KNN9120PerFieldKnnVectorsFormat.java#L94-L106
The problem is that every time a merge operation occurs, we spawn a new executor that is never shut down. We would like a `close` function to be added as an interface with default noop behavior. This function would be called whenever a merge operation is completed. That way we can shut down our merge thread pool every time it is done executing. Is this feasible?
cc @navneet1v @shatejas
Contributor guide
Research direction
Start by locating the TaskExecutor interface and the merge-operation completion path described in the issue. Review the referenced k-NN implementation in KNN9120PerFieldKnnVectorsFormat.java, especially lines 94-106, to understand how the executor is created. Done means a close hook with default no-op behavior can be invoked after merges so implementations can release their thread pools.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100