JanusGraph / JanusGraph/janusgraph
Do not fold HasContainers from HasStep into full scan if they are not used for index
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Currently `JanusGraphStep` folds `HasContainers` from following `HasSteps` in case `JanusGraphStep` is not traversed from specific vertices / edges and not using an index (i.e. full scan).
As a continuation to #3724 (issue #3244) we can improve such queries by leverage an `JanusGraphHasStep` multi-query optimization (implemented in #3724). Notice that without #3724 we won't be able to leverage that optimization because `query.batch-property-prefetch = true` is applied only to `JanusGraphVertexStep` or `JanusGraphEdgeStep`, but with #3724 we will be able to make any `has` step multi-queriable.
Here is a simple example how the next full scan query is executed right now (`graph.traversal().V().has("foo", "bar")`):
```
Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[foo.eq(bar)]) 100 100 224.643 100.00
constructGraphCentricQuery 10.972
constructGraphCentricQuery 0.070
GraphCentricQuery 216.454
\_condition=(foo = bar)
\_orders=[]
\_isFitted=false
\_isOrdered=true
\_query=[]
scan 215.883
\_query=[]
\_fullscan=true
\_condition=VERTEX
>TOTAL - - 224.643 -
```
As we can see from the above profile `HasStep` is removed and all `HasContainers` are moved under `JanusGraphStep` which doesn't leverage multi-query optimization for `HasContainer` tests. Thus, such full scan queries will fetch vertex properties one by one instead of using batch properties pre-fetching.
This logic of moving following `HasContainers` is inside `GraphCentricQueryBuilder.constructQueryWithoutProfile`.
I think as a follow up to #3724 we could improve it to not fold those `HasContainers`.
Of course, it is related to full-scan jobs only, but I assume it could improve jobs like re-indexing or other jobs which are using full-scans.
Contributor guide
Research direction
Start in GraphCentricQueryBuilder.constructQueryWithoutProfile and trace how JanusGraphStep folds HasContainers from following HasSteps during full scans. Read the JanusGraphHasStep multi-query optimization from #3724, then verify that eligible full-scan traversals keep those HasContainers in a multi-queriable HasStep rather than folding them into JanusGraphStep. Done means the full-scan behavior uses batch property prefetching without changing indexed traversal behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100