apache / apache/lucene

query caching leads to absurdly slow queries [LUCENE-9002]

Open
#10,045 14 comments 0 reactions 0 assignees View on GitHub
affects-version:7.7.2 affects-version:8.2 legacy-jira-label:cache legacy-jira-label:performance legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

**Description**
We have dozens of ES clusters(based on Lucene) for metric scenarios. Most of the queries are like this: _host_ip:10.10.10.10 AND timestamp:[2019-10-01 00:00:00 TO 2019-10-05 23:59:59]_. And we frequently encounter some absurdly slow queries.

**Solution**
For a long time range query(e.g. 5 days), each range query will consume tens of megabytes of memory and spend hundreds of milliseconds to cache, but the benefits are not obvious. And those large cache entries will cause frequent cache eviction. So it's better to skip the caching action directly when large range query appears with a selective lead iterator.

---
Migrated from [LUCENE-9002](https://issues.apache.org/jira/browse/LUCENE-9002) by Guoqiang Jiang, updated Aug 11 2021
Linked issues:
- #9075
- #9260

Contributor guide

Open the contributing guide

Research direction

Start by reading the migrated LUCENE-9002 discussion and the linked issues #9075 and #9260, then trace how long range queries enter query caching. Confirm the memory and cache-eviction behavior described for selective lead iterators, and define completion as avoiding ineffective caching without regressing query results or useful cache hits.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.