feast-dev / feast-dev/feast

DeltaLake Table Source offline store implementation

Open
#1,533 5 comments 0 reactions 1 assignee Claimed by @antoniocavalcante View on GitHub
Community Contribution Needed kind/feature kind/project priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
3d 16h
Merged PRs (30d)
15

Description

**Is your feature request related to a problem? Please describe.**
Scalable data stores—Data warehouse and modern Data Lakes—are common today for offline store implementations or as a source of data ingestion. These stores are optimized and support ACID functionality where clean data can reside, and new data constantly can be updated and merged, accommodating data changes and observing schema evolutions. This ability to ingest data from or provide an offline store as a `feast.data_source.FileSource` from these stores extends
Feast's ecosystem to modern data lakes such as [Delta Lake](https://delta.io/), [Apache Hudi](https://hudi.apache.org/), or [Apache Iceberg](https://iceberg.apache.org/).

A similar feature request for [HudiTableSource](https://github.com/feast-dev/feast/issues/1494) has been filed by @blvp

**Describe the solution you'd like**
Extend `feast.data_source.FileSource(...)` to take table names and locations to read from, for both local or remote sources.

**Describe alternatives you've considered**
I would have to save my [Delta Lake tables ](https://github.com/dmatrix/mlflow-tests/blob/master/py/delta-lake/create_weather_forecast_features.py)as a single parquet file and use that as `FileSource`, which may defeat the purpose of being able to ingest point-in-time data from these modern data lake sources.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.