gchq / gchq/sleeper

Spark support for 'and' filters

Open
#6,846 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

When I use the Spark integration to run Spark against a Sleeper table, I would like 'and' queries to be pushed to Sleeper as much as possible.

### Description / Background

If I create a system test deployment of Sleeper with the standard test data and create a Spark DataFrame as follows:

val df = spark.read.format("sleeper.spark.SleeperTableProvider").option("instanceid", "my_id").option("tablename", "vertex").load()
df.registerTempTable("my-table")

then the following returns quickly because the filters are pushed down to the source:

spark.sql("SELECT * FROM my-table WHERE key > 'a' AND key < 'ab'")

However, the following causes a full table scan:

spark.sql("SELECT * FROM my-table WHERE (key > 'a' AND key < 'ab') OR (key > 'u' AND key < 'uz')")

This is because the `pushFilter` method in `FindFiltersToPush` doesn't consider `And`s. It is not as simple as the `Or` case though because we can only push down an `Or` predicate to Sleeper if both parts can be pushed down (e.g. `key = 'a' OR key = 'b'` can be converted to 2 Sleeper queries but `key = 'a' OR value = 'b'` cannot be converted to a range query because it is necessary to look in all partitions to find rows where `value = 'b'`). But with an `And` predicate we might be able to push down just one of the predicates which means that one part of the predicate can be pushed and one part can't be pushed.

### Acceptance Criteria

The following should return quickly

spark.sql("SELECT * FROM my-table WHERE (key > 'a' AND key < 'ab') OR (key > 'u' AND key < 'uz')")

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.