feast-dev / feast-dev/feast

'feast apply' does not create online store tables in remote mode

Open
#6,693 6 comments 0 reactions 0 assignees View on GitHub
kind/bug priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Expected Behavior

With `online_store.type: remote` + `registry.registry_type: remote`, a client-side `feast apply` should create the corresponding feature view tables in the server-side online store, so that `feast materialize` can load data successfully — i.e. the schema setup done by `SqliteOnlineStore.update()` during a local `feast apply` should also take effect on the feature-server side when feature views are applied through the remote registry.

## Current Behavior

`feast apply` (remote) succeeds and registers the feature view, but **no table is created server-side**:

- After `feast apply`, `online_store.db` does not exist (only `registry_sqlite.db` is present).
- `feast materialize` reads offline data on the client, POSTs batches to the feature server's `/write-to-online-store`, and gets **HTTP 500**. The materialize command fails:

```
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=6566): Max retries exceeded with url: /write-to-online-store (Caused by ResponseError('too many 500 error responses'))
```

Feature server log shows the root cause:

```
File ".../feast/feature_store.py", line 3126, in write_to_online_store
provider.ingest_df(feature_view, df)
File ".../feast/infra/passthrough_provider.py", line 197, in online_write_batch
self.online_store.online_write_batch(config, table, data, progress)
File ".../feast/infra/online_stores/sqlite.py", line 220, in online_write_batch
conn.execute(
sqlite3.OperationalError: no such table: test_project_driver_hourly_stats
```

After the failed materialize, `online_store.db` exists but is **0 bytes with zero tables**.

Root cause (in source):

- `RemoteOnlineStore.update()` is a **no-op** (`pass`): `feast/infra/online_stores/remote.py` L706-715. `feast apply` relies on `OnlineStore.update()` to create tables, so remote apply never creates them on the server-side online store.
- `feast serve` does **not** call `SqliteOnlineStore.update()` for newly-applied feature views, so the table is never created server-side either.
- `SqliteOnlineStore.online_write_batch()` only `INSERT`s (no `CREATE TABLE`): `feast/infra/online_stores/sqlite.py` L152-239. Table creation happens only in `update()` (L291-315), which is never invoked in this flow.

Net effect: the remote online store path is unusable for materialization unless the user manually runs `feast apply` locally on the feature-server repo, defeating the purpose of the remote architecture.

## Steps to reproduce

Setup: a **server repo** (`test_project/feature_repo`) hosting registry + online store, and a **client repo** (`test_demo`) using remote registry + remote online store.

1. Server repo `test_project/feature_repo/feature_store.yaml`:

```yaml
project: test_project
registry:
registry_type: sql
path: sqlite:///data/registry_sqlite.db
provider: local
online_store:
type: sqlite
path: data/online_store.db
entity_key_serialization_version: 3
auth:
type: no_auth
```

2. Client repo `test_demo/feature_store.yaml` (remote registry + remote online store):

```yaml
project: test_project
registry:
registry_type: remote
path: localhost:6570 # feast serve_registry (gRPC)
provider: local
online_store:
type: remote
path: http://localhost:6566 # feast serve (HTTP feature server)
entity_key_serialization_version: 3
auth:
type: no_auth
```

3. Client repo `test_demo/feature_definitions.py` (a single simple FeatureView):

```python
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource, Project
from feast.types import Float32, Int64

project = Project(name="test_project", description="A project for driver statistics")
driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats_source = FileSource(
name="driver_hourly_stats_source",
path="data/driver_stats.parquet",
timestamp_field="event_timestamp",
created_timestamp_column="created",
)

driver_stats_fv = FeatureView(
name="driver_hourly_stats",
entities=[driver],
ttl=timedelta(days=1),
schema=[
Field(name="conv_rate", dtype=Float32),
Field(name="avg_daily_trips", dtype=Int64),
],
online=True,
source=driver_stats_source,
version="latest",
)
```

4. Start the servers from the server repo:

```bash
cd test_project/feature_repo
feast serve_registry --port 6570 # gRPC registry server
feast serve --port 6566 # HTTP feature server (separate shell)
```

5. From the client repo, apply (remote):

```bash
cd test_demo
feast apply
# Applying changes for project test_project
# Deploying infrastructure for driver_hourly_stats
```

6. Inspect the server-side online store — `online_store.db` does **not** exist (only `registry_sqlite.db`):

```bash
ls test_project/feature_repo/data
# registry_sqlite.db (online_store.db is absent)
```

7. From the client repo, materialize (remote):

```bash
cd test_demo
feast materialize 2021-04-12T00:00:00 2026-08-02T00:00:00
```

8. Observe the failure: client fails with `too many 500 error responses` against `/write-to-online-store`; feature server logs `sqlite3.OperationalError: no such table: test_project_driver_hourly_stats`; `online_store.db` is left as a 0-byte file with no tables.

### Specifications

- Version: Feast SDK 0.64.0 (Python 3.12.10)
- Platform: Windows 11 (10.0.26200), AMD64 — relevant code is platform-independent, expected to reproduce on Linux/macOS too
- Subsystem: `infra/online_stores/remote` (RemoteOnlineStore), `infra/online_stores/sqlite` (SqliteOnlineStore), feature server (`feast serve` / `write_to_online_store`)

Contributor guide

Open the contributing guide

Research direction

Start with RemoteOnlineStore.update() in feast/infra/online_stores/remote.py, then trace SqliteOnlineStore.update() and online_write_batch() in feast/infra/online_stores/sqlite.py and the feature-server write_to_online_store path in feast/feature_store.py. Reproduce with the remote feast apply and materialize commands; done means the server-side online_store.db and feature-view table exist after apply and materialize completes without the missing-table error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
backend-api-design, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.