MemberJunction / MemberJunction/MJ

CodeGen: entity name field is chosen by a non-deterministic LLM call then frozen permanently, so identical schemas diverge irreconcilably

Open
#3,608 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

Two related defects in `codegen-lib`'s advanced-generation pass, both found while diagnosing a live write failure in a MemberJunction Open App. Filing together because they compound: the first makes entity shape non-deterministic, and the second makes the non-determinism invisible.

All line references are `@memberjunction/codegen-lib@5.51.0`, `dist/Database/manage-metadata.js`.

---

## Defect 1 — name-field selection is an LLM call whose result is then frozen forever

`selectNameFieldWinner` (`:4866`) resolves the name field of an entity in three steps:

1. **Stability** — if exactly one currently-flagged field is still eligible, it wins *regardless of what the LLM proposed this run*. The source comment: "An already-valid single name field never drifts."
2. **Repair** — several flagged and eligible: prefer a field literally called `Name`, else first by `Sequence`.
3. **Fresh pick** — nothing flagged: take the first eligible candidate from `result.nameFields`, which is **a ranked candidate list returned by an LLM** (`ag.identifyFields(...)`, `:4673`).
4. Otherwise `null` — the entity gets no name field, and FK-name virtual columns are skipped for it.

For an entity with a column literally called `Name`, this is uneventful. For an entity **without** one, step 3 decides — and step 1 then makes that decision permanent.

The two rules interact badly:

- A database where the LLM proposed a field is **frozen on that answer forever**.
- A database where it did not keeps falling through to step 3 on **every** run, re-rolling a non-deterministic call each time.

**Neither converges toward the other, and there is no mechanism that reconciles them.** Two databases with byte-identical schema can therefore hold permanently different entity metadata, and the difference is not cosmetic: the name field determines `RelatedEntityNameFieldMap`, which determines whether *every view holding an FK to that entity* gains a derived name column.

### Observed

Entity `: Assessment Sessions`, no literal `Name` column, name field `SubjectEntityName` (`nvarchar(255)` — passes every check in `isFieldEligibleForNameField`):

| | clean installs | long-lived dev database |
|---|---|---|
| `SubjectEntityName.IsNameField` | `1` | `0` (0 of 28 fields flagged) |
| `RelatedEntityNameFieldMap` on referencing FKs | `'AssessmentSession'` | `NULL` |
| `vwAssessments` | **27 columns** | **26 columns** |

Both databases have `AutoUpdateIsNameField = 1` on every field — nothing is pinned off, and the analysis demonstrably runs (`needsFieldAnalysis` is true). The dev database has simply never been handed an eligible candidate.

### Why this is worse than cosmetic drift

A downstream consumer declares `@ResultTable` from `EntityField` and executes `INSERT INTO @ResultTable EXEC spCreate`, where the procedure returns `SELECT * FROM vw`. If a view gains a derived column that the entity's `EntityField` rows do not describe, **every write to that entity fails** with `Column name or number of supplied values does not match table definition`. Reads look fine, so nothing surfaces until a write happens.

We hit exactly that: an interview ran end to end and scored correctly, then the persist failed and produced zero rows.

### Suggested direction

- Make step 3 deterministic, or at minimum make it not the sole input — a documented convention (single bounded-text non-PK field, or an explicit config key) would remove the non-determinism entirely.
- Failing that, **log when step 3 fires and what it chose.** A run that silently changes an entity's shape should say so.
- Consider whether "no eligible candidate" should be sticky too, so an entity does not oscillate between having and not having a name field across runs.

---

## Defect 2 — `AICircuitOpen` skips the analysis with no call and no log

`processEntityAdvancedGeneration` (`:4658`):

```js
// Credential circuit tripped earlier this run — skip cleanly (no LLM call, no error log).
if (ag.AICircuitOpen) {
return;
}
```

The comment states the behaviour plainly, and the behaviour is the problem. On a credential or vendor failure the smart-field analysis is skipped for **every** entity, silently.

Combined with Defect 1 this is materially worse than it looks:

- For an already-flagged database, harmless — step 1 holds.
- For an **unflagged** one, the entity's shape stays undecided, and the run is **indistinguishable from one where the LLM ran and declined to propose anything**. There is no signal anywhere that a decision which affects generated SQL was skipped rather than made.

An operator cannot tell "CodeGen analysed this and concluded no name field" from "CodeGen never asked". Those warrant very different responses.

### Suggested direction

Emit a single warning per run when the circuit is open — how many entities were skipped, and that smart-field identification did not run. It does not need to be an error; it needs to not be silent. `CodeGen complete` currently reports success for a run in which an entire analysis phase never executed.

---

## Workaround, for anyone hitting this

Pin the flags explicitly and stop relying on inference. `applyNameFieldUpdates` respects `AutoUpdateIsNameField = 0` — the source calls those fields ones "the user owns":

```sql
UPDATE ef
SET ef.IsNameField = 1,
ef.AutoUpdateIsNameField = 0
FROM __mj.EntityField ef
INNER JOIN __mj.Entity e ON e.ID = ef.EntityID
WHERE e.Name = ''
AND ef.Name = '';
```

Key by entity **name** rather than id if your entity ids differ across installs. Setting `IsNameField` alone is not sufficient — the field stays enrolled in auto-update and a later pass can clear it.

**Any entity whose name field is not literally called `Name` should be pinned at creation.** Doing it up front is far cheaper than reconciling two databases that have already diverged, since nothing converges them automatically.

Contributor guide

Open the contributing guide

Research direction

Start in dist/Database/manage-metadata.js@5.51.0, reading selectNameFieldWinner and processEntityAdvancedGeneration, then trace applyNameFieldUpdates and the related metadata/view generation. Reproduce repeated generation for an entity without a literal Name field and a run with AICircuitOpen. Done means the name-field outcome is consistent or its choice is surfaced, and skipped analysis is reported distinctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.