MemberJunction / MemberJunction/MJ
feat(ai): add SupportsStructuredOutput vendor capability + thread into prompt execution
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Summary
Introduces a **vendor-level capability flag — `AIModelVendor.SupportsStructuredOutput`** — that marks a specific *model × inference-provider* pairing as supporting **native, provider-enforced structured output** (constrained / grammar-guided JSON decoding against a schema, e.g. OpenAI Structured Outputs / `json_schema` response formats).
This is the foundation for hardening MJ's prompt-based JSON envelope (the `LoopAgentResponse` contract and any `OutputType: object` prompt) with provider-guaranteed shape **where the provider supports it**, while keeping our current parse → JSON5 → LLM-repair → retry cascade as the universal fallback. It attacks the exact prose-drift failure mode we already invest retry machinery in — **without giving up cross-provider portability**.
It mirrors the existing `SupportsEffortLevel` / `SupportsStreaming` vendor capabilities end-to-end, so the plumbing follows an established, battle-tested pattern.
Originally targeted the **5.49** release (minor).
## What changed (on the branch)
- **Migration** (`V202607191200__v5.49.x__AIModelVendor_SupportsStructuredOutput.sql`) — adds `SupportsStructuredOutput BIT NOT NULL DEFAULT 0` to `AIModelVendor` + `sp_addextendedproperty` description. **SQL Server only.**
- **`AIPromptRunner`** (`@memberjunction/ai-prompts`) — threads the capability through:
- `ModelVendorCandidate.supportsStructuredOutput` (populated from the vendor only — **no model-level fallback**, since this is a vendor capability) and `ModelSelectionResult.vendorSupportsStructuredOutput`.
- the execution chain `executeWithValidationRetries → executeModelWithFailover → executeModel` (new optional param appended at each signature; existing call sites unaffected).
- `executeModel` sets `ChatParams.supportsStructuredOutput = true` **only when** the selected vendor supports it **AND** the effective `responseFormat` is `'JSON'` or `'ModelSpecific'`.
- **`BaseLLM`** (`@memberjunction/ai`) — new `SupportsStructuredOutput` virtual getter (code-level default `false`; DB metadata overrides — same precedent as `SupportsPrefill`).
- **`ChatParams`** (`@memberjunction/ai`) — new `supportsStructuredOutput?: boolean` hint.
- **Changeset** — minor bump for `@memberjunction/ai` and `@memberjunction/ai-prompts`.
## Design: safe-by-default, zero portability loss
The `ChatParams.supportsStructuredOutput` hint is a **capability signal, not a schema**. Drivers that implement constrained decoding can honor it (attach a provider `json_schema` response format); drivers that don't **simply ignore it**, so enabling the flag never changes behavior for a provider that lacks the capability. This preserves MJ's portability across the ~30 providers behind `BaseLLM`.
## Scope / follow-up (intentionally not in the branch)
- **PostgreSQL** — PG conversion (including registering the new boolean column in `SQLConverter`'s `CoreMetadataBooleanColumns`) is **build-engineer scope**, handled during the PG migration pass.
- **Per-provider constrained-decoding implementations** (drivers overriding `SupportsStructuredOutput` and honoring the `ChatParams` hint), starting with OpenAI.
- **Schema derivation** from a prompt's `OutputExample` into a provider `json_schema`.
- **Short-circuiting the JSON-repair/retry cascade** when structured output guarantees the shape.
- **Unit tests** for the driver behavior land with the driver implementations above.
## ⚠️ Requires CodeGen before it builds
The branch **assumes CodeGen has generated `MJAIModelVendorEntity.SupportsStructuredOutput`**. `packages/AI/Prompts` references the strongly-typed property directly and **will not compile until the migration is applied and `mj codegen` is run**. Sequence to green:
1. Apply the migration.
2. Run `mj codegen` (generates the entity property).
3. Build `@memberjunction/ai`, `@memberjunction/ai-prompts`.
No behavioral change ships until a driver implements the follow-up; existing prompt execution is unaffected (the flag defaults off for every vendor).
---
## Source
Converted from PR #3200 (deprioritized — branch preserved, PR closed).
- **Pull request:** https://github.com/MemberJunction/MJ/pull/3200
- **Branch:** [`claude/vercel-eve-agent-comparison-xpm12d`](https://github.com/MemberJunction/MJ/tree/claude/vercel-eve-agent-comparison-xpm12d)
- **Original author:** @AN-BC
The branch has **not** been deleted. To resume this work, check out `claude/vercel-eve-agent-comparison-xpm12d` and reopen a PR.
/cc @AN-BC
Contributor guide
Research direction
Start from the preserved branch and review V202607191200__v5.49.x__AIModelVendor_SupportsStructuredOutput.sql, AIPromptRunner, BaseLLM, and ChatParams. Apply the migration, run mj codegen, then build @memberjunction/ai and @memberjunction/ai-prompts. Done means the capability is threaded end to end, defaults off, and existing prompt execution remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- ai, backend-api-design, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100