MemberJunction / MemberJunction/MJ

CodeGen: processOrganicKeyConfig makes 3 DB round trips per related-entity pair — ~1hr manageMetadata phase and thousands of no-op UPDATEs

Open
#3,815 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Summary

`ManageMetadataBase.processOrganicKeyConfig()` issues **three sequential database round trips for every (organic key, related entity) pair** in `additionalSchemaInfo.json`, with no entity-resolution caching, no deduplication of the config's `RelatedEntities` array, and no change detection before writing. On a schema with a realistic number of organic keys this turns `manageMetadata` into an hour-long phase that looks like a hang, and it writes thousands of no-op `UPDATE` statements into the CodeGen migration log on every run.

Observed on **5.51.0**; the same code is present in `main` (checked at `6.1.0-edge.1`), so this is not fixed by upgrading.

## Environment

- `@memberjunction/codegen-lib` 5.51.0 (reproduced against `main` @ `6.1.0-edge.1`)
- SQL Server (Azure SQL), remote connection
- One application schema, 37 entities, organic keys generated by DBAutoDoc

## Reproduction

1. Generate an `additionalSchemaInfo.json` containing `OrganicKeys` for a schema of ~37 tables. In our case DBAutoDoc produced **131 organic keys** across 31 tables, totalling **4,143** `(key, relatedEntity)` pairs.
2. Run `mj codegen`.
3. `manageMetadata` enters `processOrganicKeyConfig` and stays there for roughly an hour.

The phase makes no progress indication distinguishable from a hang: the log is a wall of near-identical `Update organic key related entity: "" → ` lines, and because the same organic-key names recur across many owning tables (`Organization Name Match` appears on 12 different tables, `Member Id Match` on 9), it reads as though the same work is repeating.

It is not actually an infinite loop — it does terminate — but it is slow enough and repetitive enough to be indistinguishable from one, which is why we cancelled the run.

## Root cause

https://github.com/MemberJunction/MJ/blob/main/packages/CodeGenLib/src/Database/manage-metadata.ts

The inner loop of `processOrganicKeyConfig` (step 3, "Upsert EntityOrganicKeyRelatedEntity for each related entity") performs per pair:

1. `runQueryWithParams` against `vwEntities` to resolve `reConfig.TableName` → entity ID
2. `runQueryWithParams` against `EntityOrganicKeyRelatedEntity` to test existence
3. `LogSQLAndExecute` for the `UPDATE` / `INSERT`

Four separate problems compound:

**1. Entity resolution is not cached.** The `vwEntities` lookup runs once per pair even though the set of distinct related tables is tiny. In our config, 37 distinct tables are re-resolved an average of **111 times each** — 4,143 queries where 37 would do.

**2. `RelatedEntities` is not deduplicated.** The config legitimately contains repeated `(SchemaName, TableName)` entries within a single organic key's `RelatedEntities` array — DBAutoDoc emits them when a related entity is reachable by more than one path. `Members / Organization Name Match` has 8 duplicates out of 44 entries; `Carriers / Organization Name Match` has 9 of 41. Across the config that is **350 fully redundant iterations (8.4%)**, each costing three round trips and ending in a redundant `UPDATE` of a row just written.

**3. The `UPDATE` is unconditional.** Existing rows are rewritten even when every column value is identical. On a steady-state re-run this produces thousands of no-op writes. Our cancelled run logged **3,792 `UPDATE [__mj].[EntityOrganicKeyRelatedEntity]` statements against only 2,862 distinct row IDs**, all of them value-identical to what was already stored. This also churns `TrackRecordChanges` history and bloats the generated migration file (4.6 MB, 81,401 lines, of which the organic key updates are the overwhelming majority).

**4. No batching.** Every statement is its own round trip. At a typical ~250 ms RTT to Azure SQL, ~12,700 round trips is ~53 minutes — which matches what we saw.

Arithmetic for our config:

| | |
|---|---|
| Organic keys | 131 |
| `(key, relatedEntity)` pairs | 4,143 |
| Pairs after dedupe | 3,793 (**350 redundant**) |
| Distinct related tables | 37 |
| DB round trips today | **~12,700** |
| Round trips if resolution is cached, pairs deduped, existence preloaded | **~3,800** |

## Suggested fix

All four are independent and each is worthwhile on its own:

1. **Cache entity resolution.** Resolve the distinct `(SchemaName, TableName)` set once into a `Map` before the loop — or load `vwEntities` once, since `processOrganicKeyConfig` already runs after entities are created. Removes ~4,100 queries.
2. **Deduplicate `RelatedEntities`** by `(SchemaName, TableName)` when building `OrganicKeyRelatedEntityConfig[]` in `extractOrganicKeysFromConfig`. This also fixes the "last duplicate silently wins" ambiguity, where two entries for the same related entity with different `DisplayName` / `Sequence` produce order-dependent results.
3. **Preload existing `EntityOrganicKeyRelatedEntity` rows** into a `Map` keyed on `EntityOrganicKeyID + RelatedEntityID` with one query, instead of one existence probe per pair.
4. **Skip no-op updates** by comparing the incoming values against the loaded row and only issuing an `UPDATE` when something actually differs. This is the change that stops the migration-log bloat and the `TrackRecordChanges` churn.

A progress line every N pairs (`processed 400/4143 organic key relationships`) would also make the phase distinguishable from a hang while the above are in flight.

## Note on partial runs

Cancelling mid-phase appears to be safe — the writes are idempotent upserts and there is no prune/delete step for organic keys, so a re-run converges. Worth confirming that is intentional and documented, since the phase's duration makes cancellation likely.

Contributor guide

Open the contributing guide

Research direction

Start in packages/CodeGenLib/src/Database/manage-metadata.ts at ManageMetadataBase.processOrganicKeyConfig and trace extractOrganicKeysFromConfig. Run mj codegen with an additionalSchemaInfo.json containing OrganicKeys, then verify entity resolution and existing-row handling avoid redundant work, duplicate relationships are removed, and no-op updates no longer bloat the migration log.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, typescript
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.