feast-dev / feast-dev/feast

How to backfill offline store with UDF-transformed data for StreamFeatureView?

Open
#6,281 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

### Background

I'm using `StreamFeatureView` with `KafkaSource` + local provider. The real-time ingestion path works well — `StreamProcessor.ingest_stream_feature_view(to=ONLINE_AND_OFFLINE)` consumes from Kafka, applies the pandas UDF, and writes transformed features to both online and offline stores.

Now I need to **backfill historical data** that predates the streaming pipeline. I have raw Kafka history available via ETL export, but I'm not sure what the recommended approach is to get UDF-transformed features into the offline store.

### My setup

```python
@stream_feature_view(
source=kafka_source, # KafkaSource with batch_source=FileSource(parquet)
mode="pandas",
udf=my_udf, # computes derived features (e.g. aggregations, flags) from raw fields
)
def my_features(df): ...
```

### What I've tried

**1. ETL export + `materialize()`**

Export raw Kafka history to `batch_source` parquet, then call `store.materialize(start, end)`.

This reads the batch_source, applies the UDF, and writes to the **online store** — but the offline store is not updated. After `materialize()`, calling `get_historical_features()` still returns raw data without the UDF-computed derived features.

**2. Re-ingest via `StreamProcessor` from Kafka**

Use `StreamProcessor.ingest(to=ONLINE_AND_OFFLINE)` with offset reset to consume historical messages.

This only works within Kafka's retention window and doesn't scale for large history.

### Questions

1. **Is there a recommended way to backfill the offline store with UDF-transformed data?** I want the UDF defined on the StreamFeatureView to be the single source of truth, avoiding duplication in ETL scripts.

2. **Does `materialize()` intentionally skip the offline store?** If so, what is the expected workflow for keeping the offline store in sync with UDF-transformed features after a historical backfill?

3. **Batch_source and offline store are the same file** — For local/file-based offline stores, `materialize()` reads from the `batch_source` parquet, which is the same file that `write_to_offline_store()` writes to. Even if `materialize()` were extended to write back to the offline store, this would create problems:

- UDF applied twice on subsequent runs (transformed rows re-read and re-transformed)
- Data duplication (raw + transformed rows coexist)
- Idempotency broken (each run appends more duplicates)

Is separating the batch_source (raw input) from the offline store (transformed output) something that has been considered? Or is there an existing pattern I'm missing?

Any guidance or pointers would be greatly appreciated! I may well be missing something in the existing workflow.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.