MemberJunction / MemberJunction/MJ
CodeGen never prunes EntityField rows orphaned by its own base-view regeneration — saves break until an unrelated mj migrate heals them
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Summary
When a CodeGen run regenerates a base view with **fewer columns** than before — without the entity's *table* schema changing — the `EntityField` rows describing the removed columns are **never pruned by CodeGen**, on that run or any later run. The declared field count no longer matches the base-view column count, so **every save on the entity fails with Msg 213**, and CodeGen can never converge out of the state it created. The only thing that heals it is `mj migrate` (via `R__RefreshMetadata`'s unscoped `spDeleteUnneededEntityFields` call) — a different command that isn't part of the CodeGen flow.
The current trigger is the 6.1 hierarchy opt-in gate (#3939): any entity whose self-referencing FK is not opted in via `EntityField.Configuration` loses its `Root*`/`Depth`/`Path`/`IsLeaf`/`ChildCount` view columns on the next CodeGen run, while the virtual `EntityField` rows for them persist. MJ core hit exactly this as #3970 (43 orphans, fresh-install blocker); open apps hit it now, one by one, as they run CodeGen on 6.1 (`bizapps-accounting` did: `Journal Entries`, 26 declared fields vs 24 view columns). But the gap is general: **any** future change that shrinks a base view's column set without touching the table reproduces it.
## Reproduce
```bash
# Any open-app schema with a pre-6.1 self-referencing FK (ParentID-style),
# whose baseline created Root* view columns + virtual EntityField rows.
# On MJ 6.1 / next, with the field NOT opted in via Configuration:
mj codegen # view regenerated WITHOUT Root* columns; EntityField rows persist
# → every save on the entity: Msg 213 "Column name or number of supplied values
# does not match table definition."
mj codegen # run it as many times as you like — the orphans are never pruned
mj migrate # THIS heals it — R__RefreshMetadata reruns the prune unscoped
```
Probe:
```sql
SELECT e.Name, f.Name
FROM __mj.Entity e
JOIN __mj.EntityField f ON f.EntityID = e.ID
LEFT JOIN sys.columns c
ON c.object_id = OBJECT_ID(e.SchemaName + '.' + e.BaseView) AND c.name = f.Name
WHERE f.IsVirtual = 1 AND c.name IS NULL
AND OBJECT_ID(e.SchemaName + '.' + e.BaseView) IS NOT NULL;
```
## Root cause (verified at source, `next` @ `e7bba3ee13c7`)
Four links:
1. **SQL generation covers ALL included entities every run.** `sql_codegen.ts` builds `baselineEntities` from every `IncludeInAPI` entity (minus excluded schemas) and regenerates their SQL — so a base view can change (lose columns) even when the entity was never flagged new/modified.
2. **Pass 1 defers the prune.** `manage-metadata.ts:2286` calls `manageEntityFields(..., skipDeleteUnneededFields = true)` — logged as *"deferred to post-SQL pass"*. So no prune happens before or during metadata management.
3. **The post-SQL pass is scoped to new/modified entities only.** `sql_codegen.ts:345-352` builds `pass2EntityFilter = newEntityList ∪ modifiedEntityList` (unless `forceRegeneration.enabled`) and calls `manageEntityFields(..., skipDeleteUnneededFields = false, pass2EntityFilter)`. In `manage-metadata.ts:4080-4084`, an **empty filter fast-exits the entire pass** — prune included. A non-empty filter passes the entity IDs into `spDeleteUnneededEntityFields @EntityIDs=...`, so only those entities are scanned.
4. **Nothing puts a view-shrunk entity into `modifiedEntityList`.** The list is populated from *table-schema-driven* changes — new fields (`manage-metadata.ts:4999`), updated fields (`:5036`, `:5077`), and the prune's own results (`:5219`). A `Configuration` flip (or any other emission-only change) alters no table schema, so the affected entity is never flagged — the prune never even scans it, in this run or any later one.
Net: the run that removes the columns is also the run that structurally cannot see the orphans it just created, and every subsequent run has the same blind spot.
Note on `spDeleteUnneededEntityFields` itself: the sproc is **not** the problem — despite its header comment saying it removes fields "that are NOT virtual", its body has no `IsVirtual` filter; it correctly deletes any metadata field with no matching view/table column. Given an unscoped (or correctly-scoped) call it cleans this up perfectly — which is exactly why `R__RefreshMetadata` heals it. (The misleading comment is worth fixing while in there.)
## Why relying on `R__RefreshMetadata` isn't enough
- It runs only on `mj migrate`, which isn't part of the CodeGen flow — the standard dev loop is *migrate → codegen*, so the broken state created by codegen persists until the **next unrelated** migrate.
- Flows that run only app-scoped migrations never execute MJ core's repeatables at all.
- Between the codegen and the healing migrate, **every save on the entity fails** — including `MJ: Record Changes`-mediated audit writes, so in an affected core scenario (#3970) effectively all saves fail.
## Suggested fix
Any one of these closes the gap; (a) is the most surgical:
- **(a)** Track entities whose emitted base-view column set changed during Step 2 and add them to `modifiedEntityList` before the post-SQL pass — the existing scoped prune then covers them.
- **(b)** Run the post-SQL prune **unscoped** (it is a single SP call; the scoping optimization can stay for the other, per-entity steps).
- **(c)** Stop deferring the prune in pass 1 *in addition to* the scoped pass-2 prune — pass 1 unscoped catches leftovers from prior runs.
Plus the doc nit: fix `spDeleteUnneededEntityFields`' header comment ("NOT virtual") to match its actual behavior.
## Evidence
- **MJ core:** #3970's own measurements — after `mj codegen` at `e45fde5b2c`, 43 orphaned `EntityField` rows; not removed by CodeGen; retired by `R__RefreshMetadata` on migrate (per the closing verification).
- **Open app:** `bizapps-accounting`, `Journal Entries` — 26 declared fields vs 24 base-view columns after CodeGen on 6.1; persisted across subsequent CodeGen runs; required an explicit `DELETE` in the fix migration.
- Code paths above verified at `next` @ `e7bba3ee13c7` (2026-08-25).
## Related
- #3939 — the hierarchy opt-in gate (the current trigger; the gate itself is good and is not the bug).
- #3970 — MJ core's instance of this failure (closed via core migrations + `R__RefreshMetadata`; this issue is the general CodeGen-convergence gap that made #3970 possible and that every open app now hits independently).
- #3344 — PostgreSQL's `R__RefreshMetadata` never calls the prune at all, so on PG even the migrate-time heal is absent, making this gap strictly worse there.
Contributor guide
Research direction
Start with sql_codegen.ts and the pass-2 logic around lines 345-352, then trace the deferred and filtered calls in manage-metadata.ts around lines 2286 and 4080-4084. Reproduce the orphaned EntityField rows with the supplied SQL probe after a shrinking CodeGen run, and compare the behavior with spDeleteUnneededEntityFields and R__RefreshMetadata. Done means CodeGen prunes the orphan rows and saves no longer fail without requiring an unrelated migrate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100