MJ at scale: metadata/UI/CodeGen/agents all materialize every entity all-at-once — decouple cost from entity count

## Summary

MJ visibly degrades — slow CodeGen, sluggish Explorer UI, heavy startup, agent token blowups — as the **entity count** grows. Beyond the known CodeGen weakness (#2581), there is a **single architectural root cause that spans every surface**, and the existing perf work does not address it.

This issue is a diagnostic + phased roadmap to **decouple cost from entity count** (~2,000 as a ballpark design point, but architecturally unbounded).

## The fundamental diagnosis (the "one thing wrong")

MJ's existing performance work is almost entirely **micro-optimization** — shrinking the *constant factor* of per-entity work (O(1) lookup maps, lazy field-by-name indexes, cache write-path fixes, request coalescing, stale-while-revalidate, IndexedDB caching — see `plans/hotspot-perf-optimization.md`, `plans/PERFORMANCE_AUDIT.md`, `plans/startup-performance-expired-token.md`).

None of it changes the **architecture**, which is uniformly:

> **Materialize every entity, everywhere, all at once** — at build time *and* on every client, every startup.

That assumption is invisible at ~300 entities and breaks at ~2,000. The micro-opts make a broken shape cheaper; they don't change the shape. Every surface repeats the same anti-pattern:

| Surface | "All at once" manifestation |
|---|---|
| **CodeGen (build)** | Regenerates SQL/TS/forms for **all** entities every run; emits **monolithic** files (one 275K-line `entity_subclasses.ts`). |
| **Runtime metadata (startup)** | Entire `MJ_Metadata` dataset (every entity/field/relationship/permission) ships in **one JSON payload**, rebuilt into N×M in-memory objects on **every client**. |
| **UI (Explorer)** | **Every** entity form statically imported into the bundle; entity/nav/search lists rendered **unvirtualized**. |
| **Agents / MCP** | Full entity catalog and full single-entity schemas serialized into context. |

The roadmap is organized around **changing the shape** on each surface, while reusing the micro-opts already in place.

## Concrete holes, by surface (with evidence)

### A. Runtime metadata load — the epicenter
- **One giant payload:** `ProviderBase.Config()` → `GetAllMetadata()` (`packages/MJCore/src/generic/providerBase.ts:~3359`) → `GetDatasetByName('MJ_Metadata')`. Server resolver `packages/MJServer/src/resolvers/DatasetResolver.ts:51,59` runs the dataset RunViews and `JSON.stringify`s the whole result **per request** (no precompute/compression). Client `packages/GraphQLDataProvider/src/graphQLDataProvider.ts:~2110` `JSON.parse`s it whole.
- **Full reconstruction every boot:** `PostProcessEntityMetadata` (`providerBase.ts:~3445`) instantiates **every** object — `new EntityInfo(e)` per entity (`entityInfo.ts:2567`), `new EntityFieldInfo` per field (`:2597`), plus permissions/relationships/settings. At 2,116 entities × ~30 fields ≈ **60K+ constructions per boot**.
- **No subset path:** runtime **always loads all schemas** — the `IncludeSchemas/ExcludeSchemas` filter is CodeGen-only (`providerBase.ts:~3338`, explicit comment). There is no `GetEntityMetadata(name)`; it's all-or-nothing.
- **The load-bearing constraint:** the whole codebase assumes `new Metadata().Entities` is a **synchronous, complete** array — **~207** `EntityByName(...)` + **~496** `.Entities` + **~49** `EntityByID(...)` call sites. This is *why* it's all-at-once, and the central risk for any fix.
- **The async seam that saves us:** `GetEntityObject()` (`providerBase.ts:~3740`) is **already `async`** and is the *sole* factory funnel for `BaseEntity`. `RunView` is async too. The hot paths that need *full* metadata (`.Fields`) are already behind an `await` — the hook for lazy hydration.

### B. UI — Explorer
- **Eager form imports:** `packages/MJExplorer/src/app/generated/generated-forms.module.ts` statically imports/declares **290 entity form components** (15 bundled sub-modules). Grows linearly with entity count; no lazy route chunking → bundle parse cost at startup. (CodeGen chunks at `maxComponentsPerModule=25`, `packages/CodeGenLib/src/Angular/angular-codegen.ts:239`.)
- **Unvirtualized lists:** shell search loads/sorts **all** `AllowUserSearchAPI` entities into an array (`packages/Angular/Explorer/explorer-core/src/lib/shell/shell.component.ts:~2581`); nav drawer + Database Designer entity list filter **all** entities in memory with **no CDK virtual scroll**.

### C. CodeGen — build time (partly tracked by #2581)
- **Regenerates everything:** SQL gen deletes all generated `.sql` and regenerates views/SPs/permissions for **all** entities each run (`packages/CodeGenLib/src/Database/sql_codegen.ts:147-256`); the scoping (`newEntityList`/`modifiedEntityList`) only narrows the metadata-field/advanced-gen pass, **not** the main SQL loop.
- **Monolithic outputs:** single concatenated files — `entity_subclasses.ts` (104K lines core / 275K custom), `MJServer/.../generated.ts` (88K lines) — built by string concat (`entity_subclasses_codegen.ts:142-160`, `graphql_server_codegen.ts:22-43`). TS compiler + IDE degrade superlinearly.
- **Already known/planned:** #2581 (`plans/codegen-large-schema-improvements.md`) catalogs timeouts/backfill/resume on 2,116-table schemas; `plans/codegen/scoped-entity-regeneration-plan.md` designs scoped regeneration + a CodeGenReporter + E2E golden-fixture suite. **Build on these — don't restate.**

### D. Agents / MCP
- `packages/AI/MCPServer/src/Server.ts:806-841` — `Get_Entity_List` returns **all** entity names as JSON; `Get_Single_Entity` serializes a **full** EntityInfo (all fields/relationships/permissions). No pagination / lite-schema. At 2,000 entities this is large per call and explodes agent token budgets when multiple entities are described.
- `AIEngineBase` loads 25+ AI entities eagerly at startup (`plans/startup-performance-expired-token.md` Cause 4 / Task 7) — same all-at-once shape on the AI metadata side.

## What's already done or planned (do NOT re-propose)

- **Micro-opts merged/planned:** O(1) `_entityMapByName/ByID`, lazy `FieldByName` index, `LocalCacheManager` write-path fixes, request coalescing/dedup, `FastStartupMode`, stale-while-revalidate, `TrustLocalCacheCompletely`, batched IDB reads, SQLCodeGen O(n²) fix (#2508) — see `plans/hotspot-perf-optimization.md` §1.1 and `PERFORMANCE_AUDIT.md`.
- **Startup path:** token-refresh fix, cold-load cache-check skip, deferred engine tiers — `plans/startup-performance-expired-token.md`.
- **CodeGen:** scoped regeneration + reporter + E2E suite — `plans/codegen/scoped-entity-regeneration-plan.md`; large-schema resilience — #2581.

The new contribution here is the **architectural shape change** these plans don't make.

## Roadmap — change the shape, phased by risk

Ordered so each phase delivers standalone relief and de-risks the next. **Phase 0/1 are low-risk and preserve the synchronous `md.Entities` contract entirely.**

### Phase 0 — Precompute + compress the metadata payload *(Low risk, ship first)*
Attacks transport + server CPU without touching the sync contract or client reconstruction.
- Server: precompute the `MJ_Metadata` JSON once (on boot + on metadata-change invalidation) instead of `JSON.stringify` per request; serve **brotli/gzip** bytes (this JSON compresses ~8–12×). Files: `DatasetResolver.ts`, dataset build path, a cache-invalidation hook.
- Client: content-negotiate + decompress in `graphQLDataProvider.ts`.
- **Win:** removes per-request stringify CPU and multi-MB wire size at 2,000 entities. **Contract: untouched.**

### Phase 1 — Lazy-hydration plumbing in `EntityInfo` *(Low risk, no behavior change)*
Add a `_hydrated` flag + lazy children to `EntityInfo` (`Fields`/`Relationships`/`Permissions`/`Settings`) and a dev-mode guard that warns when `.Fields` is read on an unhydrated shell — **but keep loading eagerly**. Pure plumbing; flips on later. Surfaces the real offender list empirically rather than auditing ~700 call sites by hand.

### Phase 2 — Lite catalog dataset + on-demand hydration *(Medium risk)*
- New `MJ_Metadata_Catalog` dataset: entity-level columns only (ID/Name/Schema/BaseView/PK/flags), **no** fields/relationships/permissions. Build `_entityMapByName/ByID` from it → **`md.Entities` / `EntityByName` stay synchronous and complete.**
- Hydrate full per-entity metadata lazily through the **already-async** `GetEntityObject()` / `RunView` funnel (`await EnsureEntityHydrated(name)` before instance creation); cache hydrated entities. The Phase-1 dev guard catches non-funnel `.Fields` readers → gate those behind an explicit `EnsureEntityLoaded(name)`.
- **Win:** eager startup cost ∝ entity **count** (one catalog row each), not entity **size** (~20–30× fewer eager objects + bytes). CodeGen/`Refresh()` get an **opt-out "hydrate everything"** escape hatch (they need complete + full metadata).
- **Avoid sharding (per-app/per-schema):** it breaks the *completeness* of `md.Entities` (cross-app `.filter/find`, cross-shard relationship targets) with a wide silent blast radius. Catalog-shells keep completeness; only field-depth is lazy.

### Phase 3 — UI: lazy forms + virtualized lists *(Medium risk)*
- Lazy-load entity form components via Angular `loadComponent()` route chunks instead of 290 static imports in `generated-forms.module.ts` (CodeGen change in `angular-codegen.ts`). Cuts initial bundle/parse.
- CDK virtual scrolling for entity/nav/search lists (shell + Database Designer); pair with the lite catalog so the lists never need full hydration.

### Phase 4 — CodeGen shape *(folds into existing #2581 / scoped-regen plans)*
- Make the **main SQL loop** honor `newEntityList/modifiedEntityList` (not just the field pass) — true incremental regen.
- Split monolithic generated files into per-entity (or per-N) modules to tame TS compile/IDE cost.
- Pre-index `allFields` by `EntityID` to kill the O(n²) filter in `manage-metadata.ts:~4862`.

### Phase 5 — Agents / MCP lite schema *(Low/Medium risk)*
- Paginate `Get_Entity_List`; add a **lite** single-entity schema (names + types, no permissions) for `Get_Single_Entity`; context-filter the catalog to entities relevant to the task. Split `AIEngineBase` into core (always) vs extended (lazy) per `startup-performance` Task 7.

## Blast radius
- **Epicenter:** `packages/MJCore/src/generic/{providerBase.ts, entityInfo.ts, metadata.ts}` — hydration flag, lazy getters, `EnsureEntityLoaded`, catalog load.
- **Server/transport:** `MJServer/src/resolvers/DatasetResolver.ts`, `GraphQLDataProvider/src/{graphQLDataProvider.ts, storage-providers.ts}` (catalog vs full IDB stores).
- **Ordering guarantee:** `baseEntity.ts` reads `_EntityInfo.Fields` — safe only because `GetEntityObject` hydrates before `new BaseEntity` (already async).
- **Audit targets:** `BaseEngine` startup + any `md.Entities.filter` reader (catalog-only vs needs-fields); `CloneAllMetadata`/`CopyMetadataFromGlobalProvider` multi-provider clone must carry hydration state; **CodeGen must force-hydrate-all**.

## Verification (for the eventual implementation)
- **Synthetic scale harness:** seed a DB with ~2,000 entities (or point at an Aptify-shaped schema) and measure, before/after each phase: cold-start TTI, `MJ_Metadata` payload bytes (wire + decompressed), `EntityInfo` construction count/time, Explorer bundle size + first-paint, MCP `Get_Entity_List`/`Get_Single_Entity` payload bytes + agent token counts.
- **Contract safety net:** the Phase-1 dev guard must report **zero** unhydrated-`.Fields` reads after Phase-2 gating; run the existing MJCore unit suite (789 tests) + the CodeGen golden-fixture E2E suite (from `scoped-entity-regeneration-plan.md`) at each phase — **no regressions**.
- **Per-phase independence:** Phase 0 and Phase 1 must show wins / zero behavior change on a *current-size* DB too, proving they're safe to ship before the big architectural flip.

## Related
- #2581 — CodeGen large-schema operational resilience
- `plans/codegen/scoped-entity-regeneration-plan.md` — scoped CodeGen regeneration + reporter + E2E suite
- `plans/hotspot-perf-optimization.md`, `plans/PERFORMANCE_AUDIT.md`, `plans/startup-performance-expired-token.md` — prior micro-optimization work

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Start by reading plans/hotspot-perf-optimization.md, plans/PERFORMANCE_AUDIT.md, and plans/codegen/scoped-entity-regeneration-plan.md, then trace ProviderBase.Config()/GetAllMetadata(), DatasetResolver.ts, graphQLDataProvider.ts, and GetEntityObject(). The issue is an architectural roadmap spanning metadata, Explorer, CodeGen, and MCP rather than a self-contained change; done requires a chosen phase, measured scale-harness improvements, zero unhydrated-Fields reads, and passing the MJCore and CodeGen suites.

Written by the indexing model from the issue text.