JanusGraph / JanusGraph/janusgraph

Reindex times out for a big graph

Open
#1,389 8 comments 0 reactions 0 assignees View on GitHub
kind/performance
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Observed on JanusGraph 0.3.1 with Cassandra as the storage backend and Solr as the index backend.

My graph has over 14 million vertices. I added a new mixed index and I wanted to run the reindex operation as described in the official documentation ([Executing a Reindex job on JanusGraphManagement](https://docs.janusgraph.org/latest/index-admin.html#_executing_a_reindex_job_on_janusgraphmanagement)). I made sure all indexed keys are in a status which permits reindexing by calling:
```
ManagementSystem.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED, SchemaStatus.ENABLED)
```
But whenever I try to execute the reindex job (from a Java program or from the Gremlin console):
```
mgmt.updateIndex(mgmt.getGraphIndex(indexName), SchemaAction.REINDEX).get();
```
it fails after approximately 3 minutes with the following error:
```
2019-01-19 00:34:31.723 ERROR NONE 7 --- [ Thread-9] o.j.d.k.s.StandardScannerExecutor : Exception occurred during job execution: {}
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:155) [janusgraph-core-0.3.1.jar!/:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
```
followed by this repeated a few times:
```
2019-01-19 00:34:31.827 ERROR NONE 7 --- [ Thread-10] o.j.d.k.s.StandardScannerExecutor : Data-pulling thread interrupted while waiting on queue or data
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_191]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_191]
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_191]
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:340) [janusgraph-core-0.3.1.jar!/:?]
```

I ran identical code on a smaller graph (around 30 thousand vertices) and it succeeded.

I also found this thread: https://groups.google.com/forum/#!msg/janusgraph-users/9KZG8y-AdNI/lI_HubxiCQAJ - which suggests I am not the only one facing this sort of problems for big graphs.

Contributor guide

Open the contributing guide

Research direction

Start with StandardScannerExecutor.java at the reported timeout and data-pulling interruption, then reproduce mgmt.updateIndex(..., SchemaAction.REINDEX).get() against the large and small graph cases described. Trace the Cassandra-backed scan used by the reindex job and compare its behavior; done means the large-graph reindex completes without the reported timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.