MemberJunction / MemberJunction/MJ
Standardize AI model/vendor fallback — per-capability sufficiency guarantee + ModelResolver extraction
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 308
Description
## 0. The guarantee this delivers
> **For every AI capability MemberJunction uses (LLM, Embeddings, Reranker, Image Generation — and, vacuously, TTS/STT/Video), if a deployment has at least one configured vendor that supports that capability _and has a resolvable credential_, the corresponding feature works.**
The guarantee is **per-capability, not per-vendor**. A vendor that simply doesn't sell a capability (Anthropic for image generation) is an accepted gap. What is _not_ acceptable — and what MJ does in several places today — is requiring a **specific** vendor when other configured, credentialed vendors offer the same capability.
### Operator-visible symptom
> *"I configured a vendor key that is fully capable of this task, but MJ still complains as if I must supply a different, specific vendor's key."*
This is a real defect in the current tree (`next`), not a misconfiguration. It is **narrow** — the primary prompt path already does the right thing — but **real** in ~9 bypass sites, three of which fail even when a capable, credentialed alternative is configured.
---
## 1. Current-state audit (verified against `next`)
### 1.1 The primary path is already correct ✅
`AIPromptRunner.ExecutePrompt` is the gold standard we want everywhere:
- **Candidate building** — `buildModelVendorCandidates` (`packages/AI/Prompts/src/AIPromptRunner.ts` ~1787-2470).
- **Credential-aware selection** — `selectModelWithAPIKeyTracked` (~2506-2678) **filters candidates by whether a credential is actually resolvable before selecting.**
- **Cross-vendor failover** — `executeModelWithFailover` (~3019-3175); vendor-wide filtering on auth failure ~3312-3318.
- **6-level credential hierarchy** — `resolveCredentialForExecution` (~346-413): per-request `credentialId` → `AICredentialBinding` (`PromptModel`→`ModelVendor`→`Vendor`) → vendor type-default → legacy `GetAIAPIKey`.
The logic to make the guarantee true **already exists** — it is just trapped inside `AIPromptRunner` and reused by no one.
### 1.2 The bypass sites ❌
**Tier (i) — select-blind-then-fail _(the operator's exact symptom; highest severity)_** — pick the highest-priority vendor **without checking credential presence**, then hard-fail even when a capable, credentialed alternative exists:
| Site | Location |
|---|---|
| `ExecutionPlanner.selectVendorForModel` | `packages/AI/Prompts/src/ExecutionPlanner.ts:595-622` |
| `VectorSearchProvider` (embeddings) | `packages/SearchEngine/src/generic/VectorSearchProvider.ts:199-215` |
| `GenerateImageAction.prepareImageGenerator` | `packages/Actions/CoreActions/src/custom/ai/generate-image.action.ts:263-274` |
**Tier (ii) — hardcoded vendor:**
| Site | Location |
|---|---|
| Templates `{% AIPrompt %}` extension (Groq hardcoded if no model named) | `packages/Templates/engine/src/extensions/AIPrompt.extension.ts:115-134` |
| `SearchEngine.runReRanker` (scope-fixed driverClass + latent inverted-priority bug) | `packages/SearchEngine/src/generic/SearchEngine.ts:940-943` |
**Tier (iii) — key-aware but no retry-on-failure:**
| Site | Location |
|---|---|
| `AIModelRunner.findBestVendor` | `packages/AI/Prompts/src/AIModelRunner.ts:257-270` |
| `ParallelExecutionCoordinator` per-task | `packages/AI/Prompts/src/ParallelExecutionCoordinator.ts:571-580` |
| Realtime `selectRealtimeVendor` (×2) | `packages/AI/Agents/src/base-agent.ts:1781-1792`, `packages/AI/Agents/src/realtime/realtime-client-session-service.ts:1224-1235` |
**Server entrypoints (route last):** `RunAIPromptResolver.ExecuteSimplePrompt` (`packages/MJServer/src/resolvers/RunAIPromptResolver.ts` ~401-463) and its `EmbedText` mutation (~729-745). No client/React runtime makes direct LLM calls — they route through these mutations.
### 1.3 `ModelResolver` does not exist on `next`
The 2.4k-line prototype referenced in the prior issue lived only on the closed `audit/model-vendor-fallback` branch. A `feat/ai-model-resolver-phase1` worktree exists locally but is **not merged**.
---
## 2. Hard constraint to preserve
`MJAIPromptEntity.FailoverStrategy` is a **deliberate hard-fail escape hatch** (Skip-style "curated acceptable models only — error loudly if none available"). Strategies: `None`, `SameModelDifferentVendor`, `NextBestModel`, `PowerRank`. `None` ⇒ execute **only** the first candidate, no failover. The new "use any capable credentialed vendor" behavior is the **default**, not an override — `None` must keep hard-failing verbatim.
Housekeeping: `CK_AIPrompt_FailoverStrategy` has duplicated enum values (visible in the generated type). Original constraint: `migrations/v2/V202507010540__v2.62.x__Failover_Prompt_Strategy.sql`.
---
## 3. The fix: extract → reuse → retrofit
### 3.1 Package home — `@memberjunction/aiengine` (dependency-cycle-checked)
- `aiengine` is the **lowest common ancestor** of all four bypass packages (`ai-prompts`, `search-engine`, `templates`, `core-actions` already depend on it) and owns the `AIEngine.Instance` metadata cache + credential-binding helpers.
- `@memberjunction/credentials` depends only on `core`/`global`/`core-entities` — **not** on `aiengine` — so `aiengine` can add it with **no cycle**, letting the resolver call `CredentialEngine.Instance` directly.
- **Not** `ai-prompts` (too heavy/circular for `search-engine`/`templates`).
### 3.2 `ModelResolver` API (capability-generic, credential-aware, execution-agnostic)
Lives in `packages/AI/Engine/src/ModelResolver.ts`; **never imports** `BaseLLM`/`BaseEmbeddings`/`BaseReRanker`/`BaseImageGenerator` — callers supply the execute callback.
```ts
export type AICapability = 'LLM' | 'Embeddings' | 'Reranker' | 'Image';
export interface ResolvedModelCandidate {
model: MJAIModelEntityExtended; vendorId?: string; vendorName?: string;
driverClass: string; apiName?: string; priority: number;
source: 'explicit' | 'model-vendor' | 'power-rank' | 'fallback';
}
export interface ResolveOptions {
capability: AICapability; modelId?: string; modelName?: string;
vendorName?: string; // a *preferred* (not pinned) hint, e.g. legacy 'Groq'
preferredVendorId?: string; configurationId?: string; requireInferenceProvider?: boolean;
contextUser?: UserInfo; apiKeys?: { driverClass: string; apiKey: string }[];
credentialId?: string; verbose?: boolean;
}
export interface ResolveResult { candidates: ResolvedModelCandidate[]; consideredModels: ConsideredModel[]; }
export interface FailoverOptions {
strategy: 'None' | 'SameModelDifferentVendor' | 'NextBestModel' | 'PowerRank';
maxAttempts?: number; delaySeconds?: number;
errorScope?: 'All' | 'NetworkOnly' | 'RateLimitOnly' | 'ServiceErrorOnly';
}
export class ModelResolver {
// credential-FILTERED, priority-ordered candidates — the single place the guarantee is enforced
async resolveCandidates(opts: ResolveOptions): Promise;
async resolveCredential(candidate: ResolvedModelCandidate, opts: ResolveOptions): Promise;
// generic cross-vendor failover for ANY capability; strategy==='None' runs candidates[0] only
async withFailover(
candidates: ResolvedModelCandidate[], failover: FailoverOptions,
executeFn: (candidate: ResolvedModelCandidate, apiKey: string, attempt: number) => Promise,
options?: {
isRetriable?: (result: TResult) => { retriable: boolean; error?: Error }; // for drivers returning {success:false}
onAttemptFailed?: (info: FailoverAttemptInfo) => void; // AIPromptRun persistence stays in the runner
contextUser?: UserInfo;
}
): Promise;
}
```
---
## 4. Phasing
**Phase 1 — Extract (additive, zero behavior change).** Lift `resolveCredentialForExecution`, `hasCredentialsAvailable`, the `selectModelWithAPIKeyTracked` loop, and the `executeModelWithFailover` control-flow into `ModelResolver`. `AIPromptRunner` keeps prompt-aware candidate *building* and delegates execution to `ModelResolver.withFailover` with `executeModel` as `executeFn`. Add `@memberjunction/credentials` to `packages/AI/Engine/package.json`; export from its `index.ts`. **No bypass site touched.**
**Phase 1.5 — `FailoverStrategy` CHECK-constraint dedupe migration.** Re-issue `CK_AIPrompt_FailoverStrategy` canonically (`SameModelDifferentVendor`, `NextBestModel`, `PowerRank`, `None`) — **keep `None`**. Reconcile the default-value drift first (migration default `SameModelDifferentVendor` vs `getFailoverConfiguration` fallback `None`).
**Phases 2–5 — Retrofit, ordered payoff-over-risk:**
| Phase | Tier | Sites |
|---|---|---|
| 2 | (iii) lowest risk | `AIModelRunner.findBestVendor`, `ParallelExecutionCoordinator`, both realtime `selectRealtimeVendor` |
| 3 | (i) **fixes the real bug** | `ExecutionPlanner.selectVendorForModel`, `VectorSearchProvider`, `generate-image.action` |
| 4 | (ii) hardcoded + latent bugs | `AIPrompt.extension` (Groq → `vendorName:'Groq'` preferred), `SearchEngine.runReRanker` (+ fix inverted-priority) |
| 5 | server entrypoints | `RunAIPromptResolver.ExecuteSimplePrompt` / `EmbedText` |
---
## 5. Verification
- **Phase 1 proves no behavior change:** existing `AIPromptRunner.failover.test.ts` / `.model-selection.test.ts` / `.credential-errors.test.ts` / `.power-match-fallback.test.ts` pass **unchanged**. Add `packages/AI/Engine/src/__tests__/ModelResolver.test.ts`.
- **`None` escape-hatch regression (mandatory):** `withFailover(c, {strategy:'None'}, fn)` invokes `fn` **exactly once** and rethrows; `None` survives in the constraint post-1.5.
- **Per-capability sufficiency smoke (acceptance criterion):** with the highest-priority vendor's key absent but a secondary present, assert Embeddings (`VectorSearchProvider`), Image (`generate-image.action`), and Reranker (`runReRanker`) each succeed on the secondary — and reranker selects the correct highest-priority *available* vendor (inverted-priority regression). Stub `GetAIAPIKey` per `driverClass` to run credential-free.
- **Per retrofit:** `resolveCandidates` returns only credentialed candidates in priority order; failover advances past a `{success:false, canFailover:true}` first candidate.
- Build the dependency cone (`search-engine`, `core-actions`, `templates`, `MJServer`) to confirm the new `aiengine` edge introduces no break.
---
## 6. Per-capability ledger (before → after Phase 5)
| Capability | Today | After |
|---|---|---|
| **LLM** | Failover only inside `AIPromptRunner`; bypass sites pin | SATISFIED |
| **Embeddings** | Single-vendor at execution everywhere | SATISFIED |
| **Reranker** | Single-vendor + inverted-priority bug | SATISFIED (Phase 4) |
| **Image** | `Generate Image` select-blind, hard-fails on wrong vendor | SATISFIED (Phase 3 — load-bearing) |
| **TTS / STT / Video** | No runtime consumers | VACUOUSLY SATISFIED (optional CI guard) |
---
## 7. Rollout
Phase 1 (extraction) as a focused behavior-neutral PR → Phase 1.5 (migration) standalone → Phases 2-5 as separate, independently revertible PRs in order; file a child issue per phase. All changes strictly backwards-compatible.
---
_Replaces the closed draft PR #2471 / branch `audit/model-vendor-fallback`. A local `feat/ai-model-resolver-phase1` worktree has an unmerged Phase-1 prototype._
Contributor guide
Research direction
Start with packages/AI/Engine/src/ModelResolver.ts, its index.ts and package.json, then compare the credential and failover logic in AIPromptRunner.ts. Add the resolver without changing Phase 1 behavior, preserve the None strategy, and add ModelResolver.test.ts; run the existing AIPromptRunner failover, model-selection, credential-errors, and power-match-fallback tests before considering the extraction complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- ai, backend, databases, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100