JanusGraph / JanusGraph/janusgraph

Optimize ElasticSearch search queries to not always use Scroll API

Open
#1,680 7 comments 1 reaction 0 assignees View on GitHub
index/elasticsearch kind/performance
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

We should optimize search queries to not always use Scroll API (for example on queries where only single entity is needed). Search API creates a context which is stored on ElasticSearch side (for some amount of time) and which is heavy (if created too many contexts). Some search queries do not require context to be opened at all but we open the context for any search query.

Related discussion: https://groups.google.com/forum/#!topic/janusgraph-dev/7BF9Qi7dbAs

Also, possibly we can research a solution to imitate scroll usage by using search_after for queries with sorting by unique parameter / parameters.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by reading the related discussion and locating the Java search API paths that open Elasticsearch scroll contexts; compare single-entity queries with queries that need scrolling. Done should mean unnecessary contexts are avoided, with coverage showing the affected query behaviors and any search_after approach validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases, performance, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.