JanusGraph / JanusGraph/janusgraph
Reindex times out for a big graph
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Observed on JanusGraph 0.3.1 with Cassandra as the storage backend and Solr as the index backend.
My graph has over 14 million vertices. I added a new mixed index and I wanted to run the reindex operation as described in the official documentation ([Executing a Reindex job on JanusGraphManagement](https://docs.janusgraph.org/latest/index-admin.html#_executing_a_reindex_job_on_janusgraphmanagement)). I made sure all indexed keys are in a status which permits reindexing by calling:
```
ManagementSystem.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED, SchemaStatus.ENABLED)
```
But whenever I try to execute the reindex job (from a Java program or from the Gremlin console):
```
mgmt.updateIndex(mgmt.getGraphIndex(indexName), SchemaAction.REINDEX).get();
```
it fails after approximately 3 minutes with the following error:
```
2019-01-19 00:34:31.723 ERROR NONE 7 --- [ Thread-9] o.j.d.k.s.StandardScannerExecutor : Exception occurred during job execution: {}
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:155) [janusgraph-core-0.3.1.jar!/:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
```
followed by this repeated a few times:
```
2019-01-19 00:34:31.827 ERROR NONE 7 --- [ Thread-10] o.j.d.k.s.StandardScannerExecutor : Data-pulling thread interrupted while waiting on queue or data
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_191]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_191]
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_191]
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:340) [janusgraph-core-0.3.1.jar!/:?]
```
I ran identical code on a smaller graph (around 30 thousand vertices) and it succeeded.
I also found this thread: https://groups.google.com/forum/#!msg/janusgraph-users/9KZG8y-AdNI/lI_HubxiCQAJ - which suggests I am not the only one facing this sort of problems for big graphs.
Contributor guide
Research direction
Start with StandardScannerExecutor.java at the reported timeout and data-pulling interruption, then reproduce mgmt.updateIndex(..., SchemaAction.REINDEX).get() against the large and small graph cases described. Trace the Cassandra-backed scan used by the reindex job and compare its behavior; done means the large-graph reindex completes without the reported timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100