JanusGraph / JanusGraph/janusgraph

Indexes not complete

Open
#600 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

I experience incomplete mixed indexes. I tried tracing down the cause and it looks like it only occurs when performing threaded bulk transactions. While the transactions are completed successful at once or after retries, it seems the indexing is not always successful (although I do not get any errors, the vertex never indexed).
The indexes are created upfront and then I import data.
I verified the completeness of the index with the following queries:
```
gremlin> g.V().has("startDate").count()
WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [()]. For better performance, use indexes
==>2593
gremlin> g.V().has("startDate", P.lte(new java.util.Date())).count()
==>1398
gremlin> g.V().has("startDate", P.gt(new java.util.Date())).count()
==>0
```
I tested this with a build which was a few weeks old (+Cassandra 3.x and ES 5.5.2) and today with the latest master (+Cassandra 3.x and ES 5.6.2)

The commit-result of a transaction is clearly not depending on whether the indexing has succeeded, so how can I detect/handle failed indexing transactions?

I read #483 which mentions mixed indexing on ES is much slower than on non-mixed indexes. I suspect the indexing on ES is not able to keep up with the number of commits to JanusGraph and so transactions are dropped?

Contributor guide

Open the contributing guide

Research direction

Reproduce the incomplete mixed-index behavior using threaded bulk transactions with Cassandra 3.x and Elasticsearch 5.6.2, then trace transaction commits and indexing behavior to determine where successful commits can omit vertices. Compare indexed and non-indexed query results and define how failed indexing should be detected or handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, elasticsearch, java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.