holdenk / holdenk/spark-testing-base
Not allowed to query with all of partition Keys(cassandra) with Equals
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
I see this exception when I query cassandra from Unit Tests(using SharedSparkContext) but the code works fine from the base code.
**Exception** :
```
com.datastax.driver.core.exceptions.InvalidQueryException: period cannot be restricted by more than one relation if it includes an Equal
```
**Cassandra schema**
``` SQL
CREATE TABLE metricsdb.device_counts_by_hour (
period int,
dtid text,
type text,
created int,
deleted int,
PRIMARY KEY (period, dtid, type)
)
```
**Code which accesses the cassandra table**
``` Scala
val prevHourCumulativeCountsFromDb = sc.cassandraTable("keyspace_name", "table_name")
.select("dtid", "created", "deleted").where("period = " + preHour.toInt)
```
**Complete Exception Trace**
```
ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 9.0 (TID 28)
java.io.IOException: Exception during preparation of _**SELECT "dtid", "created", "deleted" FROM "metricsdb"."device_counts_by_hour" WHERE token("period") > ? AND token("period") <= ? AND period = 2016033123 ALLOW FILTERING**_: period cannot be restricted by more than one relation if it includes an Equal
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:288) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.com$datastax$spark$connector$rdd$CassandraTableScanRDD$$fetchTokenRange(CassandraTableScanRDD.scala:302) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) ~[scala-library-2.10.5.jar:na]
at com.datastax.spark.connector.util.CountingIterator.hasNext(CountingIterator.scala:12) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) ~[scala-library-2.10.5.jar:na]
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.Task.run(Task.scala:89) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: period cannot be restricted by more than one relation if it includes an Equal
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:110) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.spark.connector.cql.PreparedStatementCache$.prepareStatement(PreparedStatementCache.scala:45) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:28) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.sun.proxy.$Proxy21.prepare(Unknown Source) ~[na:na]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:274) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
... 14 common frames omitted
```
``` SQL
SELECT "dtid", "created", "deleted" FROM "keyspace_name"."table_name" WHERE token("period") > ? AND token("period") <= ? AND period = 2016033123 ALLOW FILTERING;
```
Contributor guide
Research direction
Start by reproducing the SharedSparkContext query against the reported Cassandra schema, then inspect CassandraTableScanRDD.scala around the statement creation and token-range fetch lines in the stack trace. Compare the generated CQL with the query from the issue; done means the test query executes without Cassandra rejecting the combined restrictions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, scala, spark
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100