JanusGraph / JanusGraph/janusgraph
Optimize ElasticSearch search queries to not always use Scroll API
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
We should optimize search queries to not always use Scroll API (for example on queries where only single entity is needed). Search API creates a context which is stored on ElasticSearch side (for some amount of time) and which is heavy (if created too many contexts). Some search queries do not require context to be opened at all but we open the context for any search query.
Related discussion: https://groups.google.com/forum/#!topic/janusgraph-dev/7BF9Qi7dbAs
Also, possibly we can research a solution to imitate scroll usage by using search_after for queries with sorting by unique parameter / parameters.
Contributor guide
Research direction
No file or test is named. Start by reading the related discussion and locating the Java search API paths that open Elasticsearch scroll contexts; compare single-entity queries with queries that need scrolling. Done should mean unnecessary contexts are avoided, with coverage showing the affected query behaviors and any search_after approach validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases, performance, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100