JanusGraph / JanusGraph/janusgraph

Do not fold HasContainers from HasStep into full scan if they are not used for index

Open
#3,765 0 comments 0 reactions 0 assignees View on GitHub
area/core kind/performance
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Currently `JanusGraphStep` folds `HasContainers` from following `HasSteps` in case `JanusGraphStep` is not traversed from specific vertices / edges and not using an index (i.e. full scan).
As a continuation to #3724 (issue #3244) we can improve such queries by leverage an `JanusGraphHasStep` multi-query optimization (implemented in #3724). Notice that without #3724 we won't be able to leverage that optimization because `query.batch-property-prefetch = true` is applied only to `JanusGraphVertexStep` or `JanusGraphEdgeStep`, but with #3724 we will be able to make any `has` step multi-queriable.

Here is a simple example how the next full scan query is executed right now (`graph.traversal().V().has("foo", "bar")`):
```
Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[foo.eq(bar)]) 100 100 224.643 100.00
constructGraphCentricQuery 10.972
constructGraphCentricQuery 0.070
GraphCentricQuery 216.454
\_condition=(foo = bar)
\_orders=[]
\_isFitted=false
\_isOrdered=true
\_query=[]
scan 215.883
\_query=[]
\_fullscan=true
\_condition=VERTEX
>TOTAL - - 224.643 -
```

As we can see from the above profile `HasStep` is removed and all `HasContainers` are moved under `JanusGraphStep` which doesn't leverage multi-query optimization for `HasContainer` tests. Thus, such full scan queries will fetch vertex properties one by one instead of using batch properties pre-fetching.
This logic of moving following `HasContainers` is inside `GraphCentricQueryBuilder.constructQueryWithoutProfile`.
I think as a follow up to #3724 we could improve it to not fold those `HasContainers`.
Of course, it is related to full-scan jobs only, but I assume it could improve jobs like re-indexing or other jobs which are using full-scans.

Contributor guide

Open the contributing guide

Research direction

Start in GraphCentricQueryBuilder.constructQueryWithoutProfile and trace how JanusGraphStep folds HasContainers from following HasSteps during full scans. Read the JanusGraphHasStep multi-query optimization from #3724, then verify that eligible full-scan traversals keep those HasContainers in a multi-queriable HasStep rather than folding them into JanusGraphStep. Done means the full-scan behavior uses batch property prefetching without changing indexed traversal behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.