JanusGraph / JanusGraph/janusgraph

Provide a consistent order of mixed index results

Open
#2,573 2 comments 0 reactions 0 assignees View on GitHub
index/elasticsearch kind/enhancement
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Elasticsearch search uses Lucene doc id as the tiebreaker, which is different among the same documents in different shards. This means if you have a query (assuming it hits Elasticsearch):

`g.V().has("name", "value").range(0, 10000)`

and a subsequent query

`g.V().has("name", "value").range(10000, 20000)`

there is no guarantee that results are continuous, even if there is no write request in-between at all.

Why don't we simply use `g.V().has("name", "value").range(0, 20000)`? There are a couple of reasons: 1) a single query would take too much time and our web server will simply cancel it 2) ability to fire search queries concurrently 3) avoid using too much memory at a time

We have the need to ensure a series of gremlin queries (like above) return consistent results (we don't have many write requests so we can assume documents remain unchanged in a period of time). To achieve this, we end up putting a custom unique field (ensured by application logic) in each document and patch ES6Compat/ES7Compat to use that field as the default sorting key. So far, so good.

As a general solution, I think it would be reasonable if JanusGraph can optionally document the _id field as a keyword, and uses it as a tie-breaker (sorting key).

This is related to https://github.com/JanusGraph/janusgraph/issues/1680 but from a different aspect. The solution to both might be the same, though.

Contributor guide

Open the contributing guide

Research direction

Start with the ES6Compat and ES7Compat entry points named in the issue, and inspect how Elasticsearch sorting and tie-breaking are configured. Determine how an optional document _id keyword field could provide stable ordering across shards, then identify the relevant coverage needed to confirm paginated ranges remain continuous.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases, distributed-systems, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.