Error running JanusGraph 0.3.0 OLAP Traversal (Spark 2.2 and Hadoop 2.7.5) on Scylla

I have followed the direction in https://docs.janusgraph.org/latest/hadoop-tp3.html.

Verified the using hdfs commands on the gremlin console that Hadoop is connecting to fine.

Getting this error (wondering if anybody can comment).

[Stage 0:> 16:41:50 WARN org.apache.spark.scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, 172.20.0.8, executor 0): java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.hadoop.formats.util.input.current.JanusGraphHadoopSetupImpl
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat.lambda$static$0(GiraphInputFormat.java:48)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat$RefCountedCloseable.acquire(GiraphInputFormat.java:107)
at org.janusgraph.hadoop.formats.util.GiraphRecordReader.(GiraphRecordReader.java:55)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat.createRecordReader(GiraphInputFormat.java:69)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:179)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 32 more
Caused by: org.janusgraph.core.JanusGraphException: Could not open global configuration
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:454)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1257)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:121)
at org.janusgraph.hadoop.formats.util.input.current.JanusGraphHadoopSetupImpl.(JanusGraphHadoopSetupImpl.java:52)
... 37 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureColumnFamilyExists(AstyanaxStoreManager.java:490)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.openDatabase(AstyanaxStoreManager.java:349)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.openDatabase(AstyanaxStoreManager.java:71)
at org.janusgraph.diskstorage.keycolumnvalue.KeyColumnValueStoreManager.openDatabase(KeyColumnValueStoreManager.java:43)
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:452)
... 42 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException: [host=localdatastore_db(172.20.0.3):9160, latency=26(26), attempts=1]InvalidRequestException(why:Column family system_properties already exists)
at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateColumnFamily(ThriftClusterImpl.java:240)
at com.netflix.astyanax.thrift.ThriftClusterImpl.addColumnFamily(ThriftClusterImpl.java:215)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureColumnFamilyExists(AstyanaxStoreManager.java:487)
... 46 more
Caused by: InvalidRequestException(why:Column family system_properties already exists)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result$system_add_column_family_resultStandardScheme.read(Cassandra.java:43019)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result$system_add_column_family_resultStandardScheme.read(Cassandra.java:42997)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result.read(Cassandra.java:42931)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_system_add_column_family(Cassandra.java:1522)
at org.apache.cassandra.thrift.Cassandra$Client.system_add_column_family(Cassandra.java:1509)
at com.netflix.astyanax.thrift.ThriftClusterImpl$7.internalExecute(ThriftClusterImpl.java:245)
at com.netflix.astyanax.thrift.ThriftClusterImpl$7.internalExecute(ThriftClusterImpl.java:241)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
... 54 more

16:41:50 ERROR org.apache.spark.scheduler.TaskSetManager - Task 1 in stage 0.0 failed 4 times; aborting job
16:41:51 WARN org.apache.spark.scheduler.TaskSetManager - Lost task 3.3 in stage 0.0 (TID 11, 172.20.0.8, executor 0): TaskKilled (stage cancelled)
(0 + 2) / 257]org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 10, 172.20.0.8, executor 0): java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.hadoop.formats.util.input.current.JanusGraphHadoopSetupImpl
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat.lambda$static$0(GiraphInputFormat.java:48)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat$RefCountedCloseable.acquire(GiraphInputFormat.java:107)
at org.janusgraph.hadoop.formats.util.GiraphRecordReader.(GiraphRecordReader.java:55)
at org.janusgraph.hadoop.formats.util.GiraphInputFormat.createRecordReader(GiraphInputFormat.java:69)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:179)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 32 more
Caused by: org.janusgraph.core.JanusGraphException: Could not open global configuration
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:454)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1257)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:121)
at org.janusgraph.hadoop.formats.util.input.current.JanusGraphHadoopSetupImpl.(JanusGraphHadoopSetupImpl.java:52)
... 37 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureColumnFamilyExists(AstyanaxStoreManager.java:490)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.openDatabase(AstyanaxStoreManager.java:349)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.openDatabase(AstyanaxStoreManager.java:71)
at org.janusgraph.diskstorage.keycolumnvalue.KeyColumnValueStoreManager.openDatabase(KeyColumnValueStoreManager.java:43)
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:452)
... 42 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException: [host=localdatastore_db(172.20.0.3):9160, latency=3(3), attempts=1]InvalidRequestException(why:Column family system_properties already exists)
at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateColumnFamily(ThriftClusterImpl.java:240)
at com.netflix.astyanax.thrift.ThriftClusterImpl.addColumnFamily(ThriftClusterImpl.java:215)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureColumnFamilyExists(AstyanaxStoreManager.java:487)
... 46 more
Caused by: InvalidRequestException(why:Column family system_properties already exists)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result$system_add_column_family_resultStandardScheme.read(Cassandra.java:43019)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result$system_add_column_family_resultStandardScheme.read(Cassandra.java:42997)
at org.apache.cassandra.thrift.Cassandra$system_add_column_family_result.read(Cassandra.java:42931)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_system_add_column_family(Cassandra.java:1522)
at org.apache.cassandra.thrift.Cassandra$Client.system_add_column_family(Cassandra.java:1509)
at com.netflix.astyanax.thrift.ThriftClusterImpl$7.internalExecute(ThriftClusterImpl.java:245)
at com.netflix.astyanax.thrift.ThriftClusterImpl$7.internalExecute(ThriftClusterImpl.java:241)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
... 54 more

Start with the linked Hadoop TP3 documentation and the JanusGraphHadoopSetupImpl stack-trace entry point. Trace the failure through Backend.getStandaloneGlobalConfiguration and AstyanaxStoreManager.ensureColumnFamilyExists, then reproduce the OLAP traversal with the stated Spark, Hadoop, and Scylla setup. Done means the traversal starts without the system_properties column-family error.

Written by the indexing model from the issue text.