feast-dev / feast-dev/feast

Deprecate support for SQL queries as part of data sources

Open
#1,688 13 comments 2 reactions 0 assignees View on GitHub
keep-open kind/techdebt priority/p1
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

In the current Feast API it is possible for users to specify either a `table reference` or `query` as part of data sources
* [BigQuery](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/data_source.py#L130)
* [Redshift](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/data_source.py#L927)

**Query**
```
BigQuerySource(
query="SELECT timestamp as ts, created, f1, f2 FROM `my_project.my_dataset.my_features`",
)
```

**Table Ref**
```
BigQuerySource(
table_ref="my_project.my_dataset.my_features"
)
```

The motivation for supporting `query` was to allow users to manipulate source data prior to reading it into Feast during materialization or training dataset building. The assumption is that not all users own their sources, so they may not be able to make a change to the table schema. Or they may not have permissions to create views in their offline store.

However, we are unsure whether this assumption holds up. It seems like most users only need
1. Field mapping
2. Column projection
3. Filtering of the source rows

Supporting `query` comes with major downsides
1. It's wasteful. Feast has to execute the full query in order to do simple operations like get metadata (column names/types)
2. Feast is unable to optimize the underlying query and is forced to execute it as-is, sometimes repeatedly. This can lead to increased costs for users.
3. It requires the Feast team to maintain two separate code paths for functionality that is almost identical.
4. It makes the Feast API more complicated. We need to explain to users that they should use `table ref` if they want more optimized queries.

All of the required functionality above can be added as configuration options as part of `table ref`, which would free us up from having to support two separate means of querying offline data. I'm hoping to hear from users whether continuing to support `query` is important, or whether we can try to support similar functionality with `table ref`.

cc @MattDelac @mavysavydav

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.