MemberJunction / MemberJunction/MJ
CodeGen: processOrganicKeyConfig makes 3 DB round trips per related-entity pair — ~1hr manageMetadata phase and thousands of no-op UPDATEs
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Summary
`ManageMetadataBase.processOrganicKeyConfig()` issues **three sequential database round trips for every (organic key, related entity) pair** in `additionalSchemaInfo.json`, with no entity-resolution caching, no deduplication of the config's `RelatedEntities` array, and no change detection before writing. On a schema with a realistic number of organic keys this turns `manageMetadata` into an hour-long phase that looks like a hang, and it writes thousands of no-op `UPDATE` statements into the CodeGen migration log on every run.
Observed on **5.51.0**; the same code is present in `main` (checked at `6.1.0-edge.1`), so this is not fixed by upgrading.
## Environment
- `@memberjunction/codegen-lib` 5.51.0 (reproduced against `main` @ `6.1.0-edge.1`)
- SQL Server (Azure SQL), remote connection
- One application schema, 37 entities, organic keys generated by DBAutoDoc
## Reproduction
1. Generate an `additionalSchemaInfo.json` containing `OrganicKeys` for a schema of ~37 tables. In our case DBAutoDoc produced **131 organic keys** across 31 tables, totalling **4,143** `(key, relatedEntity)` pairs.
2. Run `mj codegen`.
3. `manageMetadata` enters `processOrganicKeyConfig` and stays there for roughly an hour.
The phase makes no progress indication distinguishable from a hang: the log is a wall of near-identical `Update organic key related entity: "" → ` lines, and because the same organic-key names recur across many owning tables (`Organization Name Match` appears on 12 different tables, `Member Id Match` on 9), it reads as though the same work is repeating.
It is not actually an infinite loop — it does terminate — but it is slow enough and repetitive enough to be indistinguishable from one, which is why we cancelled the run.
## Root cause
https://github.com/MemberJunction/MJ/blob/main/packages/CodeGenLib/src/Database/manage-metadata.ts
The inner loop of `processOrganicKeyConfig` (step 3, "Upsert EntityOrganicKeyRelatedEntity for each related entity") performs per pair:
1. `runQueryWithParams` against `vwEntities` to resolve `reConfig.TableName` → entity ID
2. `runQueryWithParams` against `EntityOrganicKeyRelatedEntity` to test existence
3. `LogSQLAndExecute` for the `UPDATE` / `INSERT`
Four separate problems compound:
**1. Entity resolution is not cached.** The `vwEntities` lookup runs once per pair even though the set of distinct related tables is tiny. In our config, 37 distinct tables are re-resolved an average of **111 times each** — 4,143 queries where 37 would do.
**2. `RelatedEntities` is not deduplicated.** The config legitimately contains repeated `(SchemaName, TableName)` entries within a single organic key's `RelatedEntities` array — DBAutoDoc emits them when a related entity is reachable by more than one path. `Members / Organization Name Match` has 8 duplicates out of 44 entries; `Carriers / Organization Name Match` has 9 of 41. Across the config that is **350 fully redundant iterations (8.4%)**, each costing three round trips and ending in a redundant `UPDATE` of a row just written.
**3. The `UPDATE` is unconditional.** Existing rows are rewritten even when every column value is identical. On a steady-state re-run this produces thousands of no-op writes. Our cancelled run logged **3,792 `UPDATE [__mj].[EntityOrganicKeyRelatedEntity]` statements against only 2,862 distinct row IDs**, all of them value-identical to what was already stored. This also churns `TrackRecordChanges` history and bloats the generated migration file (4.6 MB, 81,401 lines, of which the organic key updates are the overwhelming majority).
**4. No batching.** Every statement is its own round trip. At a typical ~250 ms RTT to Azure SQL, ~12,700 round trips is ~53 minutes — which matches what we saw.
Arithmetic for our config:
| | |
|---|---|
| Organic keys | 131 |
| `(key, relatedEntity)` pairs | 4,143 |
| Pairs after dedupe | 3,793 (**350 redundant**) |
| Distinct related tables | 37 |
| DB round trips today | **~12,700** |
| Round trips if resolution is cached, pairs deduped, existence preloaded | **~3,800** |
## Suggested fix
All four are independent and each is worthwhile on its own:
1. **Cache entity resolution.** Resolve the distinct `(SchemaName, TableName)` set once into a `Map` before the loop — or load `vwEntities` once, since `processOrganicKeyConfig` already runs after entities are created. Removes ~4,100 queries.
2. **Deduplicate `RelatedEntities`** by `(SchemaName, TableName)` when building `OrganicKeyRelatedEntityConfig[]` in `extractOrganicKeysFromConfig`. This also fixes the "last duplicate silently wins" ambiguity, where two entries for the same related entity with different `DisplayName` / `Sequence` produce order-dependent results.
3. **Preload existing `EntityOrganicKeyRelatedEntity` rows** into a `Map` keyed on `EntityOrganicKeyID + RelatedEntityID` with one query, instead of one existence probe per pair.
4. **Skip no-op updates** by comparing the incoming values against the loaded row and only issuing an `UPDATE` when something actually differs. This is the change that stops the migration-log bloat and the `TrackRecordChanges` churn.
A progress line every N pairs (`processed 400/4143 organic key relationships`) would also make the phase distinguishable from a hang while the above are in flight.
## Note on partial runs
Cancelling mid-phase appears to be safe — the writes are idempotent upserts and there is no prune/delete step for organic keys, so a re-run converges. Worth confirming that is intentional and documented, since the phase's duration makes cancellation likely.
Contributor guide
Research direction
Start in packages/CodeGenLib/src/Database/manage-metadata.ts at ManageMetadataBase.processOrganicKeyConfig and trace extractOrganicKeysFromConfig. Run mj codegen with an additionalSchemaInfo.json containing OrganicKeys, then verify entity resolution and existing-row handling avoid redundant work, duplicate relationships are removed, and no-op updates no longer bloat the migration log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100