Spark support for 'and' filters
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
When I use the Spark integration to run Spark against a Sleeper table, I would like 'and' queries to be pushed to Sleeper as much as possible.
### Description / Background
If I create a system test deployment of Sleeper with the standard test data and create a Spark DataFrame as follows:
val df = spark.read.format("sleeper.spark.SleeperTableProvider").option("instanceid", "my_id").option("tablename", "vertex").load()
df.registerTempTable("my-table")
then the following returns quickly because the filters are pushed down to the source:
spark.sql("SELECT * FROM my-table WHERE key > 'a' AND key < 'ab'")
However, the following causes a full table scan:
spark.sql("SELECT * FROM my-table WHERE (key > 'a' AND key < 'ab') OR (key > 'u' AND key < 'uz')")
This is because the `pushFilter` method in `FindFiltersToPush` doesn't consider `And`s. It is not as simple as the `Or` case though because we can only push down an `Or` predicate to Sleeper if both parts can be pushed down (e.g. `key = 'a' OR key = 'b'` can be converted to 2 Sleeper queries but `key = 'a' OR value = 'b'` cannot be converted to a range query because it is necessary to look in all partitions to find rows where `value = 'b'`). But with an `And` predicate we might be able to push down just one of the predicates which means that one part of the predicate can be pushed and one part can't be pushed.
### Acceptance Criteria
The following should return quickly
spark.sql("SELECT * FROM my-table WHERE (key > 'a' AND key < 'ab') OR (key > 'u' AND key < 'uz')")
Contributor guide
Assessment
This issue has not been assessed yet.