JanusGraph / JanusGraph/janusgraph
Indexes not complete
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
I experience incomplete mixed indexes. I tried tracing down the cause and it looks like it only occurs when performing threaded bulk transactions. While the transactions are completed successful at once or after retries, it seems the indexing is not always successful (although I do not get any errors, the vertex never indexed).
The indexes are created upfront and then I import data.
I verified the completeness of the index with the following queries:
```
gremlin> g.V().has("startDate").count()
WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [()]. For better performance, use indexes
==>2593
gremlin> g.V().has("startDate", P.lte(new java.util.Date())).count()
==>1398
gremlin> g.V().has("startDate", P.gt(new java.util.Date())).count()
==>0
```
I tested this with a build which was a few weeks old (+Cassandra 3.x and ES 5.5.2) and today with the latest master (+Cassandra 3.x and ES 5.6.2)
The commit-result of a transaction is clearly not depending on whether the indexing has succeeded, so how can I detect/handle failed indexing transactions?
I read #483 which mentions mixed indexing on ES is much slower than on non-mixed indexes. I suspect the indexing on ES is not able to keep up with the number of commits to JanusGraph and so transactions are dropped?
Contributor guide
Research direction
Reproduce the incomplete mixed-index behavior using threaded bulk transactions with Cassandra 3.x and Elasticsearch 5.6.2, then trace transaction commits and indexing behavior to determine where successful commits can omit vertices. Compare indexed and non-indexed query results and define how failed indexing should be detected or handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, elasticsearch, java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100