Data Farm — Centralized Entity Data Layer & Realtime Sync

# Data Farm — Centralized Entity Data Layer & Realtime Sync

**Parent:** [Epic #6519 — Scalable Frontend Architecture & Data Flow](https://github.com/FlowFuse/flowfuse/issues/6519)

## Problem

Entity data (teams, applications, instances, devices) is fetched ad-hoc inside components and held as local state — no single source of truth. As a result:

- An MCP tool success (e.g. "create an instance" from expert chat) doesn't update the UI — the entity exists server-side but the list can't learn about it without a reload, which loses the chat.
- The same entity is fetched at multiple levels (list + detail + status) and the copies diverge.
- Loading is a scatter of independent spinners with no coherent state.

## Solution

One Pinia store per entity (`data-farm-`) owns that entity's data. The flow is unidirectional — the epic's "Golden Path":

```text
Component → data-farm store action → API client / service → store state
▲ │
└──────────────────── reactive read (getters) ──────────────────────┘
▲
subscriber (MQTT/WS) ─────┘ pushes realtime CRUD + status into the same state
```

- **Stores own data** — components call store actions and read getters, never API clients directly.
- **Subscribers update stores** — the subscriber pattern (`team-channel.subscriber.ts` → `live-status.ts`) is generalized from status-only to full CRUD, so an MCP-triggered create/update/delete flows into the store and every mounted component reflects it.
- **Backend publishes lifecycle events** — alongside the existing `app.comms.team` emits, add `created`/`updated`/`deleted` publishers called from mutation routes.
- **Components go dumb + get loaders** — render store state; show a loader while the entity is hydrating. Loading is centralized in `UXStore`, not per-component spinners:
- **Blocking (global app loader)** — gates the shell until mandatory data hydrates: auth/user, settings, RBAC, team. Team lives here, so Teams needs no store-level loader.
- **Non-blocking (shell stays interactive)** — Page / Modal / Component loaders. For route-lazy entities the store exposes hydration flags and the component drives the shared Page Loader (`ff-page` renders `UXStore.pageLoader`).

## Relationship to Epic #6519

| Epic sub-issue | Relationship |
| --- | --- |
| **#6522 Modular State with Pinia** (✅ closed) | Prerequisite. Data-farm builds on it. |
| **#6520 Hydrate-Before-Route** (open) | Data-farm stores are what hydration populates; the loader tiers above are #6520's "Tiered Hydration" orchestration. Keep loader components shared. |
| **#6521 GBAC / router guards** (open) | Guards validate permissions against pre-loaded store data — that data is data-farm. |
| Epic concepts #3 / #5 / #7 | This sub-issue implements them for real entities. |

Net-new (not in the epic): wiring subscribers to entity CRUD, not just status — enabled by existing infra (`subscribers/`, `mqtt.service.ts`, `mqtt.transport.ts`, `comms/`).

## PR shape per entity

Tasks map 1:1 to PRs, each independently mergeable — no PR ships a store nothing reads.

| PR | Contains |
| --- | --- |
| Backend | lifecycle publisher call sites + ACL (independent) |
| Store + first consumer | store + unit tests, primary list migrated, CRUD `upsert`/`remove` — **fixes the same-session bug** |
| Subscriber | route CRUD topics → `applyRealtime` — adds **cross-session** sync |
| Remaining vue | migrate rest off local fetching, add loaders, delete dead code |

Order per entity: store first (backend in parallel) → subscriber + remaining-vue.

## Entities (child sub-issues)

| # | Entity | Notes |
| --- | --- | --- |
| 1 | **Applications** | Pilot — defines the reusable pattern and fixes the same-session bug. 4 PRs. |
| 2 | Teams | Foundational + lowest-risk. Topics already exist → no backend PR (3 PRs). |
| 3 | Hosted Instances | Status already live; entity data not centralized. Reuses the pattern. |
| 4 | Remote Instances (Devices) | Polymorphic ownership; worst scattering. Reuses the pattern. |

Future waves (same shape): Snapshots · Pipelines · Device Groups · Members & Invitations · Team Library · Blueprints · Broker topics · Tables.