apache / apache/lucene

Hook for merge thread pool

Open
#15,935 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

In opensearch k-NN, we want merge operations to be executed with a dynamic number of threads. To do this, we are giving a TaskExecutor a dynamic number of threads here: https://github.com/opensearch-project/k-NN/blob/e327675aa7c404ab0cac533a826bca0febb53fdd/src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/KNN9120PerFieldKnnVectorsFormat.java#L94-L106

The problem is that every time a merge operation occurs, we spawn a new executor that is never shut down. We would like a `close` function to be added as an interface with default noop behavior. This function would be called whenever a merge operation is completed. That way we can shut down our merge thread pool every time it is done executing. Is this feasible?

cc @navneet1v @shatejas

Contributor guide

Open the contributing guide

Research direction

Start by locating the TaskExecutor interface and the merge-operation completion path described in the issue. Review the referenced k-NN implementation in KNN9120PerFieldKnnVectorsFormat.java, especially lines 94-106, to understand how the executor is created. Done means a close hook with default no-op behavior can be invoked after merges so implementations can release their thread pools.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.