DeltaLake Table Source offline store implementation
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 15
Description
**Is your feature request related to a problem? Please describe.**
Scalable data stores—Data warehouse and modern Data Lakes—are common today for offline store implementations or as a source of data ingestion. These stores are optimized and support ACID functionality where clean data can reside, and new data constantly can be updated and merged, accommodating data changes and observing schema evolutions. This ability to ingest data from or provide an offline store as a `feast.data_source.FileSource` from these stores extends
Feast's ecosystem to modern data lakes such as [Delta Lake](https://delta.io/), [Apache Hudi](https://hudi.apache.org/), or [Apache Iceberg](https://iceberg.apache.org/).
A similar feature request for [HudiTableSource](https://github.com/feast-dev/feast/issues/1494) has been filed by @blvp
**Describe the solution you'd like**
Extend `feast.data_source.FileSource(...)` to take table names and locations to read from, for both local or remote sources.
**Describe alternatives you've considered**
I would have to save my [Delta Lake tables ](https://github.com/dmatrix/mlflow-tests/blob/master/py/delta-lake/create_weather_forecast_features.py)as a single parquet file and use that as `FileSource`, which may defeat the purpose of being able to ingest point-in-time data from these modern data lake sources.
Contributor guide
Assessment
This issue has not been assessed yet.