JanusGraph / JanusGraph/janusgraph
Provide a consistent order of mixed index results
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Elasticsearch search uses Lucene doc id as the tiebreaker, which is different among the same documents in different shards. This means if you have a query (assuming it hits Elasticsearch):
`g.V().has("name", "value").range(0, 10000)`
and a subsequent query
`g.V().has("name", "value").range(10000, 20000)`
there is no guarantee that results are continuous, even if there is no write request in-between at all.
Why don't we simply use `g.V().has("name", "value").range(0, 20000)`? There are a couple of reasons: 1) a single query would take too much time and our web server will simply cancel it 2) ability to fire search queries concurrently 3) avoid using too much memory at a time
We have the need to ensure a series of gremlin queries (like above) return consistent results (we don't have many write requests so we can assume documents remain unchanged in a period of time). To achieve this, we end up putting a custom unique field (ensured by application logic) in each document and patch ES6Compat/ES7Compat to use that field as the default sorting key. So far, so good.
As a general solution, I think it would be reasonable if JanusGraph can optionally document the _id field as a keyword, and uses it as a tie-breaker (sorting key).
This is related to https://github.com/JanusGraph/janusgraph/issues/1680 but from a different aspect. The solution to both might be the same, though.
Contributor guide
Research direction
Start with the ES6Compat and ES7Compat entry points named in the issue, and inspect how Elasticsearch sorting and tie-breaking are configured. Determine how an optional document _id keyword field could provide stable ordering across shards, then identify the relevant coverage needed to confirm paginated ranges remain continuous.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases, distributed-systems, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100