apache / apache/lucene

Modal Strategy In Concurrent Query Execution [LUCENE-9066]

Open
#10,108 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When executing a query concurrently today, we do not take any sort of system statistics into account. For e.g. if the node is under high pressure, it is not advisable to spawn a large number of threads for a query since they are most likely to be blocked waiting for CPU to be available. However, the converse is that for a lightly loaded cluster, the query can consume as many threads as required.

 

This Jira tracks high level efforts in this direction. The first idea is to account the Executor's wait queue's size as a factor when allocating slices to a query's segments.

---
Migrated from [LUCENE-9066](https://issues.apache.org/jira/browse/LUCENE-9066) by Atri Sharma (@atris), updated Dec 02 2019
Linked issues:
- #10117
- #10116

Contributor guide

Open the contributing guide

Research direction

Start with linked issues #10117 and #10116, then trace the concurrent query execution paths involving the Executor wait queue, segment slices, and thread allocation. The issue describes a high-level direction rather than a specific file, test, or acceptance criterion, so the intended behavior and completion criteria need to be established before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.