holdenk / holdenk/spark-testing-base

Not allowed to query with all of partition Keys(cassandra) with Equals

Open
#101 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

I see this exception when I query cassandra from Unit Tests(using SharedSparkContext) but the code works fine from the base code.

**Exception** :

```
com.datastax.driver.core.exceptions.InvalidQueryException: period cannot be restricted by more than one relation if it includes an Equal
```

**Cassandra schema**

``` SQL
CREATE TABLE metricsdb.device_counts_by_hour (
period int,
dtid text,
type text,
created int,
deleted int,
PRIMARY KEY (period, dtid, type)
)
```

**Code which accesses the cassandra table**

``` Scala
val prevHourCumulativeCountsFromDb = sc.cassandraTable("keyspace_name", "table_name")
.select("dtid", "created", "deleted").where("period = " + preHour.toInt)
```

**Complete Exception Trace**

```
ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 9.0 (TID 28)
java.io.IOException: Exception during preparation of _**SELECT "dtid", "created", "deleted" FROM "metricsdb"."device_counts_by_hour" WHERE token("period") > ? AND token("period") <= ? AND period = 2016033123 ALLOW FILTERING**_: period cannot be restricted by more than one relation if it includes an Equal
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:288) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.com$datastax$spark$connector$rdd$CassandraTableScanRDD$$fetchTokenRange(CassandraTableScanRDD.scala:302) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) ~[scala-library-2.10.5.jar:na]
at com.datastax.spark.connector.util.CountingIterator.hasNext(CountingIterator.scala:12) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) ~[scala-library-2.10.5.jar:na]
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.scheduler.Task.run(Task.scala:89) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: period cannot be restricted by more than one relation if it includes an Equal
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:110) ~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.spark.connector.cql.PreparedStatementCache$.prepareStatement(PreparedStatementCache.scala:45) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:28) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
at com.sun.proxy.$Proxy21.prepare(Unknown Source) ~[na:na]
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:274) ~[spark-cassandra-connector_2.10-1.6.0-M2.jar:1.6.0-M2]
... 14 common frames omitted
```

``` SQL
SELECT "dtid", "created", "deleted" FROM "keyspace_name"."table_name" WHERE token("period") > ? AND token("period") <= ? AND period = 2016033123 ALLOW FILTERING;
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the SharedSparkContext query against the reported Cassandra schema, then inspect CassandraTableScanRDD.scala around the statement creation and token-range fetch lines in the stack trace. Compare the generated CQL with the query from the issue; done means the test query executes without Cassandra rejecting the combined restrictions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, scala, spark
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.