MemberJunction / MemberJunction/MJ

feat(ai): add SupportsStructuredOutput vendor capability + thread into prompt execution

Open
#3,202 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Summary

Introduces a **vendor-level capability flag — `AIModelVendor.SupportsStructuredOutput`** — that marks a specific *model × inference-provider* pairing as supporting **native, provider-enforced structured output** (constrained / grammar-guided JSON decoding against a schema, e.g. OpenAI Structured Outputs / `json_schema` response formats).

This is the foundation for hardening MJ's prompt-based JSON envelope (the `LoopAgentResponse` contract and any `OutputType: object` prompt) with provider-guaranteed shape **where the provider supports it**, while keeping our current parse → JSON5 → LLM-repair → retry cascade as the universal fallback. It attacks the exact prose-drift failure mode we already invest retry machinery in — **without giving up cross-provider portability**.

It mirrors the existing `SupportsEffortLevel` / `SupportsStreaming` vendor capabilities end-to-end, so the plumbing follows an established, battle-tested pattern.

Originally targeted the **5.49** release (minor).

## What changed (on the branch)

- **Migration** (`V202607191200__v5.49.x__AIModelVendor_SupportsStructuredOutput.sql`) — adds `SupportsStructuredOutput BIT NOT NULL DEFAULT 0` to `AIModelVendor` + `sp_addextendedproperty` description. **SQL Server only.**
- **`AIPromptRunner`** (`@memberjunction/ai-prompts`) — threads the capability through:
- `ModelVendorCandidate.supportsStructuredOutput` (populated from the vendor only — **no model-level fallback**, since this is a vendor capability) and `ModelSelectionResult.vendorSupportsStructuredOutput`.
- the execution chain `executeWithValidationRetries → executeModelWithFailover → executeModel` (new optional param appended at each signature; existing call sites unaffected).
- `executeModel` sets `ChatParams.supportsStructuredOutput = true` **only when** the selected vendor supports it **AND** the effective `responseFormat` is `'JSON'` or `'ModelSpecific'`.
- **`BaseLLM`** (`@memberjunction/ai`) — new `SupportsStructuredOutput` virtual getter (code-level default `false`; DB metadata overrides — same precedent as `SupportsPrefill`).
- **`ChatParams`** (`@memberjunction/ai`) — new `supportsStructuredOutput?: boolean` hint.
- **Changeset** — minor bump for `@memberjunction/ai` and `@memberjunction/ai-prompts`.

## Design: safe-by-default, zero portability loss

The `ChatParams.supportsStructuredOutput` hint is a **capability signal, not a schema**. Drivers that implement constrained decoding can honor it (attach a provider `json_schema` response format); drivers that don't **simply ignore it**, so enabling the flag never changes behavior for a provider that lacks the capability. This preserves MJ's portability across the ~30 providers behind `BaseLLM`.

## Scope / follow-up (intentionally not in the branch)

- **PostgreSQL** — PG conversion (including registering the new boolean column in `SQLConverter`'s `CoreMetadataBooleanColumns`) is **build-engineer scope**, handled during the PG migration pass.
- **Per-provider constrained-decoding implementations** (drivers overriding `SupportsStructuredOutput` and honoring the `ChatParams` hint), starting with OpenAI.
- **Schema derivation** from a prompt's `OutputExample` into a provider `json_schema`.
- **Short-circuiting the JSON-repair/retry cascade** when structured output guarantees the shape.
- **Unit tests** for the driver behavior land with the driver implementations above.

## ⚠️ Requires CodeGen before it builds

The branch **assumes CodeGen has generated `MJAIModelVendorEntity.SupportsStructuredOutput`**. `packages/AI/Prompts` references the strongly-typed property directly and **will not compile until the migration is applied and `mj codegen` is run**. Sequence to green:

1. Apply the migration.
2. Run `mj codegen` (generates the entity property).
3. Build `@memberjunction/ai`, `@memberjunction/ai-prompts`.

No behavioral change ships until a driver implements the follow-up; existing prompt execution is unaffected (the flag defaults off for every vendor).

---

## Source

Converted from PR #3200 (deprioritized — branch preserved, PR closed).

- **Pull request:** https://github.com/MemberJunction/MJ/pull/3200
- **Branch:** [`claude/vercel-eve-agent-comparison-xpm12d`](https://github.com/MemberJunction/MJ/tree/claude/vercel-eve-agent-comparison-xpm12d)
- **Original author:** @AN-BC

The branch has **not** been deleted. To resume this work, check out `claude/vercel-eve-agent-comparison-xpm12d` and reopen a PR.

/cc @AN-BC

Contributor guide

Open the contributing guide

Research direction

Start from the preserved branch and review V202607191200__v5.49.x__AIModelVendor_SupportsStructuredOutput.sql, AIPromptRunner, BaseLLM, and ChatParams. Apply the migration, run mj codegen, then build @memberjunction/ai and @memberjunction/ai-prompts. Done means the capability is threaded end to end, defaults off, and existing prompt execution remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, typescript
Domain
ai, backend-api-design, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.