RegistryServer silently loses commit=False mutations when an uncached read interleaves; remote feast apply nondeterministically drops objects
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
`feast apply` against a remote registry (`registry_type: remote` -> `feast serve_registry` with a file-backed registry) persists all applied objects, or fails loudly.
## Current Behavior
Applied objects are **silently lost** whenever any `allow_cache=False` read reaches the registry server between the batched `commit=False` mutations and the final `Commit` RPC. The client exits 0; the registry on disk is missing an arbitrary subset of the batch.
In production this manifests as nondeterministic partial persistence: in our cluster (feast 0.64.0 client and server, feast-operator deployment, file-backed registry served over gRPC), repeated `feast apply` runs kept some deletions but never persisted new DataSources/FeatureViews or path updates - the registry stayed weeks stale while every apply reported success. Which mutations survive depends on the timing of concurrent readers (UI, online server, other SDK clients), i.e. it is a race.
## Root cause
Three pieces interact:
1. `FeatureStore.apply()` sends **every** mutation with `commit=False` and one final `commit()` (sdk/python/feast/feature_store.py, the `_apply_diffs`/apply flow around lines 1500-1624 in v0.64.0). For a `RemoteRegistry` these become individual gRPC calls: `ApplyDataSource(commit=False)`, ..., `Commit()`.
2. Server-side, `commit=False` mutations exist **only** in the file-backed `Registry.cached_registry_proto` (in memory, uncommitted).
3. `Registry._get_registry_proto(..., allow_cache=False)` (sdk/python/feast/infra/registry/registry.py, `registry_proto = self._registry_store.get_registry_proto(); self.cached_registry_proto = registry_proto`) **replaces the in-memory proto from disk**, silently discarding all pending uncommitted mutations.
Any uncached read between steps 1 and the final `Commit` therefore wipes the pending batch. Uncached reads are routine: `list_*`/`get_*` default to `allow_cache=False`, and the RegistryServer is a shared server - the Feast UI, an online feature server, or any other SDK client can trigger the reload at any time. The permission-check getters inside the Apply/Delete handlers themselves (`assert_permissions_to_update(getter=self.proxied_registry.get_data_source, ...)`) can also contribute.
The transaction model (accumulate uncommitted state in a shared, reload-on-read cache; commit at the end) is only sound for a single-process SDK client that owns the Registry object. Exposed behind a multi-client gRPC server, it loses writes by design.
## Steps to reproduce
Deterministic MRE, stock `feast==0.64.0`, no Kubernetes required - one interleaved uncached read is the only variable:
```python
"""RegistryServer silently discards uncommitted (commit=False) mutations
when an allow_cache=False read interleaves before the final Commit RPC.
Run: python repro.py /tmp/feast-mre-state # bug case
Control: python repro.py /tmp/feast-mre-state --no-read # no interleaved read
"""
import subprocess
import sys
import time
from pathlib import Path
from feast import Entity, FeatureStore, RepoConfig
from feast.infra.registry.remote import RemoteRegistry, RemoteRegistryConfig
from feast.value_type import ValueType
BASE = Path(sys.argv[1])
BASE.mkdir(parents=True, exist_ok=True)
REG_FILE = BASE / "registry.db"
PORT = 6570
server_config = RepoConfig(
project="mre",
provider="local",
registry=str(REG_FILE),
entity_key_serialization_version=3,
)
FeatureStore(config=server_config).apply([]) # seed project + registry.db
(BASE / "feature_store.yaml").write_text(
f"project: mre\nprovider: local\nregistry: {REG_FILE}\n"
"entity_key_serialization_version: 3\n"
)
server = subprocess.Popen(
["feast", "-c", str(BASE), "serve_registry", "-p", str(PORT)], cwd=BASE
)
time.sleep(8)
try:
remote = RemoteRegistry(
registry_config=RemoteRegistryConfig(
registry_type="remote", path=f"localhost:{PORT}"
),
project="mre",
repo_path=None,
)
ent_a = Entity(name="entity_a", join_keys=["a_id"], value_type=ValueType.STRING)
ent_b = Entity(name="entity_b", join_keys=["b_id"], value_type=ValueType.STRING)
# what `feast apply` does: batched mutations with commit=False
remote.apply_entity(ent_a, project="mre", commit=False)
# what any concurrent reader does (UI / online server / other client):
if "--no-read" not in sys.argv:
remote.list_entities(project="mre", allow_cache=False)
remote.apply_entity(ent_b, project="mre", commit=False)
remote.commit()
time.sleep(1)
names = sorted(e.name for e in FeatureStore(config=server_config).list_entities())
print("entities on disk after commit:", names)
finally:
server.terminate()
server.wait(timeout=10)
```
Output:
```
# bug case (with the interleaved uncached read):
entities on disk after commit: ['entity_b'] # entity_a silently lost
# control (--no-read):
entities on disk after commit: ['entity_a', 'entity_b']
```
## Specifications
- Version: 0.64.0 (client and server; also reads identically in current master)
- Platform: reproduced on macOS (MRE) and Kubernetes/feast-operator (production)
- Subsystem: registry server (gRPC), file-backed registry
## Possible Solution
Options, roughly in order of increasing scope:
1. **Write-through on the server**: `RegistryServer` forces `commit=True` on every mutation RPC regardless of `request.commit` when the backing registry is file-based (per-RPC durability; the client's trailing `Commit` becomes a no-op). Simple, removes the loss window, at the cost of one file write per object.
2. Guard the reload: `_get_registry_proto` refuses to discard a proto that has uncommitted mutations (dirty flag) - reads serve the dirty cache until commit.
3. Per-client staging on the server (true transaction semantics for the batched apply protocol).
4. At minimum: document that file-backed remote registries are not safe for concurrent use with writers, and recommend the SQL registry for `serve_registry` deployments.
Related but distinct: #4271 (remote `apply_materialization` failing, wontfix), #5791 (stale UI reads after delete - read-side symptom of the same cache model).
---
*Disclaimer: this investigation was AI-assisted and this issue text was written by an AI agent (reviewed by a human before filing). The MRE and its output were executed and verified as shown against stock `feast==0.64.0`.*
Contributor guide
Research direction
Start by running the deterministic MRE from the issue with and without --no-read. Then inspect sdk/python/feast/feature_store.py around the _apply_diffs/apply flow and sdk/python/feast/infra/registry/registry.py, especially _get_registry_proto and cached_registry_proto. Done means an interleaved allow_cache=False read no longer causes committed mutations to disappear, and the reproduction preserves both entities.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100