JanusGraph / JanusGraph/janusgraph
Permanent failure in storage backend CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:315)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
We are seeing the following stacktrace when trying to ingest data on JanusGraph 0.2.0. We have cassandra, elasticsearch and janusgraph run on the same local system.
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:95)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:143)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:200)
at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:737)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)
at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.acceptRecord(VertexLoaderWorker.java:109)
at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.access$000(VertexLoaderWorker.java:35)
at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker$1.accept(VertexLoaderWorker.java:130)
at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker$1.accept(VertexLoaderWorker.java:126)
at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:899)
at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.run(VertexLoaderWorker.java:126)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftKeyColumnValueStore.convertException(CassandraThriftKeyColumnValueStore.java:263)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:315)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:79)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:98)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:95)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 17 more
Caused by: TimedOutException(acknowledged_by:0, acknowledged_by_batchlog:true)
at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29624)
at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29592)
at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result.read(Cassandra.java:29526)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_atomic_batch_mutate(Cassandra.java:1108)
at org.apache.cassandra.thrift.Cassandra$Client.atomic_batch_mutate(Cassandra.java:1094)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:310)
... 22 more
Please find attached the janusgraph configuration.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at CassandraThriftStoreManager.mutateMany and CassandraThriftKeyColumnValueStore.convertException, then review the attached JanusGraph configuration and reproduce ingestion against the local Cassandra setup. The stack trace ends in Cassandra's TimedOutException; done requires identifying the cause of that timeout and verifying a documented or tested resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, elasticsearch, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100