MemberJunction / MemberJunction/MJ

CodeGen never prunes EntityField rows orphaned by its own base-view regeneration — saves break until an unrelated mj migrate heals them

Open
#4,050 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Summary

When a CodeGen run regenerates a base view with **fewer columns** than before — without the entity's *table* schema changing — the `EntityField` rows describing the removed columns are **never pruned by CodeGen**, on that run or any later run. The declared field count no longer matches the base-view column count, so **every save on the entity fails with Msg 213**, and CodeGen can never converge out of the state it created. The only thing that heals it is `mj migrate` (via `R__RefreshMetadata`'s unscoped `spDeleteUnneededEntityFields` call) — a different command that isn't part of the CodeGen flow.

The current trigger is the 6.1 hierarchy opt-in gate (#3939): any entity whose self-referencing FK is not opted in via `EntityField.Configuration` loses its `Root*`/`Depth`/`Path`/`IsLeaf`/`ChildCount` view columns on the next CodeGen run, while the virtual `EntityField` rows for them persist. MJ core hit exactly this as #3970 (43 orphans, fresh-install blocker); open apps hit it now, one by one, as they run CodeGen on 6.1 (`bizapps-accounting` did: `Journal Entries`, 26 declared fields vs 24 view columns). But the gap is general: **any** future change that shrinks a base view's column set without touching the table reproduces it.

## Reproduce

```bash
# Any open-app schema with a pre-6.1 self-referencing FK (ParentID-style),
# whose baseline created Root* view columns + virtual EntityField rows.
# On MJ 6.1 / next, with the field NOT opted in via Configuration:
mj codegen # view regenerated WITHOUT Root* columns; EntityField rows persist
# → every save on the entity: Msg 213 "Column name or number of supplied values
# does not match table definition."
mj codegen # run it as many times as you like — the orphans are never pruned
mj migrate # THIS heals it — R__RefreshMetadata reruns the prune unscoped
```

Probe:

```sql
SELECT e.Name, f.Name
FROM __mj.Entity e
JOIN __mj.EntityField f ON f.EntityID = e.ID
LEFT JOIN sys.columns c
ON c.object_id = OBJECT_ID(e.SchemaName + '.' + e.BaseView) AND c.name = f.Name
WHERE f.IsVirtual = 1 AND c.name IS NULL
AND OBJECT_ID(e.SchemaName + '.' + e.BaseView) IS NOT NULL;
```

## Root cause (verified at source, `next` @ `e7bba3ee13c7`)

Four links:

1. **SQL generation covers ALL included entities every run.** `sql_codegen.ts` builds `baselineEntities` from every `IncludeInAPI` entity (minus excluded schemas) and regenerates their SQL — so a base view can change (lose columns) even when the entity was never flagged new/modified.

2. **Pass 1 defers the prune.** `manage-metadata.ts:2286` calls `manageEntityFields(..., skipDeleteUnneededFields = true)` — logged as *"deferred to post-SQL pass"*. So no prune happens before or during metadata management.

3. **The post-SQL pass is scoped to new/modified entities only.** `sql_codegen.ts:345-352` builds `pass2EntityFilter = newEntityList ∪ modifiedEntityList` (unless `forceRegeneration.enabled`) and calls `manageEntityFields(..., skipDeleteUnneededFields = false, pass2EntityFilter)`. In `manage-metadata.ts:4080-4084`, an **empty filter fast-exits the entire pass** — prune included. A non-empty filter passes the entity IDs into `spDeleteUnneededEntityFields @EntityIDs=...`, so only those entities are scanned.

4. **Nothing puts a view-shrunk entity into `modifiedEntityList`.** The list is populated from *table-schema-driven* changes — new fields (`manage-metadata.ts:4999`), updated fields (`:5036`, `:5077`), and the prune's own results (`:5219`). A `Configuration` flip (or any other emission-only change) alters no table schema, so the affected entity is never flagged — the prune never even scans it, in this run or any later one.

Net: the run that removes the columns is also the run that structurally cannot see the orphans it just created, and every subsequent run has the same blind spot.

Note on `spDeleteUnneededEntityFields` itself: the sproc is **not** the problem — despite its header comment saying it removes fields "that are NOT virtual", its body has no `IsVirtual` filter; it correctly deletes any metadata field with no matching view/table column. Given an unscoped (or correctly-scoped) call it cleans this up perfectly — which is exactly why `R__RefreshMetadata` heals it. (The misleading comment is worth fixing while in there.)

## Why relying on `R__RefreshMetadata` isn't enough

- It runs only on `mj migrate`, which isn't part of the CodeGen flow — the standard dev loop is *migrate → codegen*, so the broken state created by codegen persists until the **next unrelated** migrate.
- Flows that run only app-scoped migrations never execute MJ core's repeatables at all.
- Between the codegen and the healing migrate, **every save on the entity fails** — including `MJ: Record Changes`-mediated audit writes, so in an affected core scenario (#3970) effectively all saves fail.

## Suggested fix

Any one of these closes the gap; (a) is the most surgical:

- **(a)** Track entities whose emitted base-view column set changed during Step 2 and add them to `modifiedEntityList` before the post-SQL pass — the existing scoped prune then covers them.
- **(b)** Run the post-SQL prune **unscoped** (it is a single SP call; the scoping optimization can stay for the other, per-entity steps).
- **(c)** Stop deferring the prune in pass 1 *in addition to* the scoped pass-2 prune — pass 1 unscoped catches leftovers from prior runs.

Plus the doc nit: fix `spDeleteUnneededEntityFields`' header comment ("NOT virtual") to match its actual behavior.

## Evidence

- **MJ core:** #3970's own measurements — after `mj codegen` at `e45fde5b2c`, 43 orphaned `EntityField` rows; not removed by CodeGen; retired by `R__RefreshMetadata` on migrate (per the closing verification).
- **Open app:** `bizapps-accounting`, `Journal Entries` — 26 declared fields vs 24 base-view columns after CodeGen on 6.1; persisted across subsequent CodeGen runs; required an explicit `DELETE` in the fix migration.
- Code paths above verified at `next` @ `e7bba3ee13c7` (2026-08-25).

## Related

- #3939 — the hierarchy opt-in gate (the current trigger; the gate itself is good and is not the bug).
- #3970 — MJ core's instance of this failure (closed via core migrations + `R__RefreshMetadata`; this issue is the general CodeGen-convergence gap that made #3970 possible and that every open app now hits independently).
- #3344 — PostgreSQL's `R__RefreshMetadata` never calls the prune at all, so on PG even the migrate-time heal is absent, making this gap strictly worse there.

Contributor guide

Open the contributing guide

Research direction

Start with sql_codegen.ts and the pass-2 logic around lines 345-352, then trace the deferred and filtered calls in manage-metadata.ts around lines 2286 and 4080-4084. Reproduce the orphaned EntityField rows with the supplied SQL probe after a shrinking CodeGen run, and compare the behavior with spDeleteUnneededEntityFields and R__RefreshMetadata. Done means CodeGen prunes the orphan rows and saves no longer fail without requiring an unrelated migrate.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.