feast-dev / feast-dev/feast

RegistryServer silently loses commit=False mutations when an uncached read interleaves; remote feast apply nondeterministically drops objects

Open
#6,663 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Expected Behavior

`feast apply` against a remote registry (`registry_type: remote` -> `feast serve_registry` with a file-backed registry) persists all applied objects, or fails loudly.

## Current Behavior

Applied objects are **silently lost** whenever any `allow_cache=False` read reaches the registry server between the batched `commit=False` mutations and the final `Commit` RPC. The client exits 0; the registry on disk is missing an arbitrary subset of the batch.

In production this manifests as nondeterministic partial persistence: in our cluster (feast 0.64.0 client and server, feast-operator deployment, file-backed registry served over gRPC), repeated `feast apply` runs kept some deletions but never persisted new DataSources/FeatureViews or path updates - the registry stayed weeks stale while every apply reported success. Which mutations survive depends on the timing of concurrent readers (UI, online server, other SDK clients), i.e. it is a race.

## Root cause

Three pieces interact:

1. `FeatureStore.apply()` sends **every** mutation with `commit=False` and one final `commit()` (sdk/python/feast/feature_store.py, the `_apply_diffs`/apply flow around lines 1500-1624 in v0.64.0). For a `RemoteRegistry` these become individual gRPC calls: `ApplyDataSource(commit=False)`, ..., `Commit()`.
2. Server-side, `commit=False` mutations exist **only** in the file-backed `Registry.cached_registry_proto` (in memory, uncommitted).
3. `Registry._get_registry_proto(..., allow_cache=False)` (sdk/python/feast/infra/registry/registry.py, `registry_proto = self._registry_store.get_registry_proto(); self.cached_registry_proto = registry_proto`) **replaces the in-memory proto from disk**, silently discarding all pending uncommitted mutations.

Any uncached read between steps 1 and the final `Commit` therefore wipes the pending batch. Uncached reads are routine: `list_*`/`get_*` default to `allow_cache=False`, and the RegistryServer is a shared server - the Feast UI, an online feature server, or any other SDK client can trigger the reload at any time. The permission-check getters inside the Apply/Delete handlers themselves (`assert_permissions_to_update(getter=self.proxied_registry.get_data_source, ...)`) can also contribute.

The transaction model (accumulate uncommitted state in a shared, reload-on-read cache; commit at the end) is only sound for a single-process SDK client that owns the Registry object. Exposed behind a multi-client gRPC server, it loses writes by design.

## Steps to reproduce

Deterministic MRE, stock `feast==0.64.0`, no Kubernetes required - one interleaved uncached read is the only variable:

```python
"""RegistryServer silently discards uncommitted (commit=False) mutations
when an allow_cache=False read interleaves before the final Commit RPC.

Run: python repro.py /tmp/feast-mre-state # bug case
Control: python repro.py /tmp/feast-mre-state --no-read # no interleaved read
"""

import subprocess
import sys
import time
from pathlib import Path

from feast import Entity, FeatureStore, RepoConfig
from feast.infra.registry.remote import RemoteRegistry, RemoteRegistryConfig
from feast.value_type import ValueType

BASE = Path(sys.argv[1])
BASE.mkdir(parents=True, exist_ok=True)
REG_FILE = BASE / "registry.db"
PORT = 6570

server_config = RepoConfig(
project="mre",
provider="local",
registry=str(REG_FILE),
entity_key_serialization_version=3,
)
FeatureStore(config=server_config).apply([]) # seed project + registry.db

(BASE / "feature_store.yaml").write_text(
f"project: mre\nprovider: local\nregistry: {REG_FILE}\n"
"entity_key_serialization_version: 3\n"
)
server = subprocess.Popen(
["feast", "-c", str(BASE), "serve_registry", "-p", str(PORT)], cwd=BASE
)
time.sleep(8)

try:
remote = RemoteRegistry(
registry_config=RemoteRegistryConfig(
registry_type="remote", path=f"localhost:{PORT}"
),
project="mre",
repo_path=None,
)
ent_a = Entity(name="entity_a", join_keys=["a_id"], value_type=ValueType.STRING)
ent_b = Entity(name="entity_b", join_keys=["b_id"], value_type=ValueType.STRING)

# what `feast apply` does: batched mutations with commit=False
remote.apply_entity(ent_a, project="mre", commit=False)

# what any concurrent reader does (UI / online server / other client):
if "--no-read" not in sys.argv:
remote.list_entities(project="mre", allow_cache=False)

remote.apply_entity(ent_b, project="mre", commit=False)
remote.commit()
time.sleep(1)

names = sorted(e.name for e in FeatureStore(config=server_config).list_entities())
print("entities on disk after commit:", names)
finally:
server.terminate()
server.wait(timeout=10)
```

Output:

```
# bug case (with the interleaved uncached read):
entities on disk after commit: ['entity_b'] # entity_a silently lost

# control (--no-read):
entities on disk after commit: ['entity_a', 'entity_b']
```

## Specifications

- Version: 0.64.0 (client and server; also reads identically in current master)
- Platform: reproduced on macOS (MRE) and Kubernetes/feast-operator (production)
- Subsystem: registry server (gRPC), file-backed registry

## Possible Solution

Options, roughly in order of increasing scope:

1. **Write-through on the server**: `RegistryServer` forces `commit=True` on every mutation RPC regardless of `request.commit` when the backing registry is file-based (per-RPC durability; the client's trailing `Commit` becomes a no-op). Simple, removes the loss window, at the cost of one file write per object.
2. Guard the reload: `_get_registry_proto` refuses to discard a proto that has uncommitted mutations (dirty flag) - reads serve the dirty cache until commit.
3. Per-client staging on the server (true transaction semantics for the batched apply protocol).
4. At minimum: document that file-backed remote registries are not safe for concurrent use with writers, and recommend the SQL registry for `serve_registry` deployments.

Related but distinct: #4271 (remote `apply_materialization` failing, wontfix), #5791 (stale UI reads after delete - read-side symptom of the same cache model).

---

*Disclaimer: this investigation was AI-assisted and this issue text was written by an AI agent (reviewed by a human before filing). The MRE and its output were executed and verified as shown against stock `feast==0.64.0`.*

Contributor guide

Open the contributing guide

Research direction

Start by running the deterministic MRE from the issue with and without --no-read. Then inspect sdk/python/feast/feature_store.py around the _apply_diffs/apply flow and sdk/python/feast/infra/registry/registry.py, especially _get_registry_proto and cached_registry_proto. Done means an interleaved allow_cache=False read no longer causes committed mutations to disappear, and the reproduction preserves both entities.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.