'feast apply' does not create online store tables in remote mode
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
With `online_store.type: remote` + `registry.registry_type: remote`, a client-side `feast apply` should create the corresponding feature view tables in the server-side online store, so that `feast materialize` can load data successfully — i.e. the schema setup done by `SqliteOnlineStore.update()` during a local `feast apply` should also take effect on the feature-server side when feature views are applied through the remote registry.
## Current Behavior
`feast apply` (remote) succeeds and registers the feature view, but **no table is created server-side**:
- After `feast apply`, `online_store.db` does not exist (only `registry_sqlite.db` is present).
- `feast materialize` reads offline data on the client, POSTs batches to the feature server's `/write-to-online-store`, and gets **HTTP 500**. The materialize command fails:
```
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=6566): Max retries exceeded with url: /write-to-online-store (Caused by ResponseError('too many 500 error responses'))
```
Feature server log shows the root cause:
```
File ".../feast/feature_store.py", line 3126, in write_to_online_store
provider.ingest_df(feature_view, df)
File ".../feast/infra/passthrough_provider.py", line 197, in online_write_batch
self.online_store.online_write_batch(config, table, data, progress)
File ".../feast/infra/online_stores/sqlite.py", line 220, in online_write_batch
conn.execute(
sqlite3.OperationalError: no such table: test_project_driver_hourly_stats
```
After the failed materialize, `online_store.db` exists but is **0 bytes with zero tables**.
Root cause (in source):
- `RemoteOnlineStore.update()` is a **no-op** (`pass`): `feast/infra/online_stores/remote.py` L706-715. `feast apply` relies on `OnlineStore.update()` to create tables, so remote apply never creates them on the server-side online store.
- `feast serve` does **not** call `SqliteOnlineStore.update()` for newly-applied feature views, so the table is never created server-side either.
- `SqliteOnlineStore.online_write_batch()` only `INSERT`s (no `CREATE TABLE`): `feast/infra/online_stores/sqlite.py` L152-239. Table creation happens only in `update()` (L291-315), which is never invoked in this flow.
Net effect: the remote online store path is unusable for materialization unless the user manually runs `feast apply` locally on the feature-server repo, defeating the purpose of the remote architecture.
## Steps to reproduce
Setup: a **server repo** (`test_project/feature_repo`) hosting registry + online store, and a **client repo** (`test_demo`) using remote registry + remote online store.
1. Server repo `test_project/feature_repo/feature_store.yaml`:
```yaml
project: test_project
registry:
registry_type: sql
path: sqlite:///data/registry_sqlite.db
provider: local
online_store:
type: sqlite
path: data/online_store.db
entity_key_serialization_version: 3
auth:
type: no_auth
```
2. Client repo `test_demo/feature_store.yaml` (remote registry + remote online store):
```yaml
project: test_project
registry:
registry_type: remote
path: localhost:6570 # feast serve_registry (gRPC)
provider: local
online_store:
type: remote
path: http://localhost:6566 # feast serve (HTTP feature server)
entity_key_serialization_version: 3
auth:
type: no_auth
```
3. Client repo `test_demo/feature_definitions.py` (a single simple FeatureView):
```python
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource, Project
from feast.types import Float32, Int64
project = Project(name="test_project", description="A project for driver statistics")
driver = Entity(name="driver", join_keys=["driver_id"])
driver_stats_source = FileSource(
name="driver_hourly_stats_source",
path="data/driver_stats.parquet",
timestamp_field="event_timestamp",
created_timestamp_column="created",
)
driver_stats_fv = FeatureView(
name="driver_hourly_stats",
entities=[driver],
ttl=timedelta(days=1),
schema=[
Field(name="conv_rate", dtype=Float32),
Field(name="avg_daily_trips", dtype=Int64),
],
online=True,
source=driver_stats_source,
version="latest",
)
```
4. Start the servers from the server repo:
```bash
cd test_project/feature_repo
feast serve_registry --port 6570 # gRPC registry server
feast serve --port 6566 # HTTP feature server (separate shell)
```
5. From the client repo, apply (remote):
```bash
cd test_demo
feast apply
# Applying changes for project test_project
# Deploying infrastructure for driver_hourly_stats
```
6. Inspect the server-side online store — `online_store.db` does **not** exist (only `registry_sqlite.db`):
```bash
ls test_project/feature_repo/data
# registry_sqlite.db (online_store.db is absent)
```
7. From the client repo, materialize (remote):
```bash
cd test_demo
feast materialize 2021-04-12T00:00:00 2026-08-02T00:00:00
```
8. Observe the failure: client fails with `too many 500 error responses` against `/write-to-online-store`; feature server logs `sqlite3.OperationalError: no such table: test_project_driver_hourly_stats`; `online_store.db` is left as a 0-byte file with no tables.
### Specifications
- Version: Feast SDK 0.64.0 (Python 3.12.10)
- Platform: Windows 11 (10.0.26200), AMD64 — relevant code is platform-independent, expected to reproduce on Linux/macOS too
- Subsystem: `infra/online_stores/remote` (RemoteOnlineStore), `infra/online_stores/sqlite` (SqliteOnlineStore), feature server (`feast serve` / `write_to_online_store`)
Contributor guide
Research direction
Start with RemoteOnlineStore.update() in feast/infra/online_stores/remote.py, then trace SqliteOnlineStore.update() and online_write_batch() in feast/infra/online_stores/sqlite.py and the feature-server write_to_online_store path in feast/feature_store.py. Reproduce with the remote feast apply and materialize commands; done means the server-side online_store.db and feature-view table exist after apply and materialize completes without the missing-table error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- backend-api-design, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100