apache / apache/drill

Unexpected behavior with Cassandra connector

Open
#2,595 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Java
Stars
2k
Forks
990
Avg merge
1d 8h
Merged PRs (30d)
5

Description

**Describe the bug**
We are using Apache Drill to add ANSI SQL capabilities to cassandra, but when using the '>', '<' or 'IN' operators when filtering data, the query plan switch from a CassandraFilter to a regular Filter, meaning all the cassandra table data is scanned, fetched, then filtered, which is not the expected behavior, as the Apache Calcite plugin supports those operators.
This results in very slow queries, and high resources consumptions.

**Screenshots**

*(boitier_id, libelle, unite and periode are keys)*

*Expected behavior (using a CassandraFilter)*
![image](https://user-images.githubusercontent.com/62694436/178532574-c276851a-cc23-46fa-8ba9-6f04cebf2658.png)
![image](https://user-images.githubusercontent.com/62694436/178532335-e20d4486-d4d5-4337-8cec-6ed6c178051e.png)

*Unexpected behavior (When using lt, gt operator)*
![image](https://user-images.githubusercontent.com/62694436/178535991-30dc3731-00d8-4a9d-8765-88d10dc2df4a.png)
![image](https://user-images.githubusercontent.com/62694436/178532850-7cfef8b1-7699-4532-af3e-602f0bc4eb37.png)

*Possible solution to get a correct behavior, but not completely, as the whole dataset for (76, '3dProd_C1','W') is loaded insted of just the portion which we would like to use*
![image](https://user-images.githubusercontent.com/62694436/178672629-4d4a5c38-fd02-40b0-aa58-cbc4504ebbe3.png)
![image](https://user-images.githubusercontent.com/62694436/178534184-265eca54-4d7e-401b-80fb-b4d7d8cdd575.png)

**Expected behavior**
The Query should use a CassandraFilter in order to fetch data efficiently, even when using '>', '<' operators, and not use a normal filter, which requires to fetch all the data from the queried table.

Should our question not belong here, feel free to remove it, but please point us to where we could ask it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Apache Drill Cassandra connector query plans described in the issue, comparing CassandraFilter with regular Filter for >, <, and IN predicates. Trace how these operators are translated through the Cassandra connector and Apache Calcite integration. Done means the plan retains CassandraFilter and fetches only the requested portion instead of scanning the whole table.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.