MemberJunction / MemberJunction/MJ
Explore native tool calling in BaseLLM and the agent framework
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Summary
MJ's LLM providers do not use the native tool/function-calling APIs that every major vendor now exposes. Agent actions are described as text inside the system prompt, and the model is asked to reply with a JSON `LoopAgentResponse` envelope that the framework parses and dispatches.
That design is portable and works on models with no tool support at all, which is a real strength. But newer, more agentically-trained models are beginning to fight it: they recognise the action contract in the prompt, attempt a native tool call, and the request declares no tools — so the provider discards the response.
This issue proposes exploring native tool calling as an **opt-in capability**, not a replacement.
## Current state
- No chat driver passes `tools` / `functionDeclarations` / `tool_choice`. Checked: OpenAI, Anthropic, Gemini, Groq, Cerebras, Mistral, xAI, OpenRouter, Vertex. (The one hit in the Gemini package is `geminiRealtime.ts`, a separate path.)
- `packages/AI/Core/src/generic/baseLLM.ts` on `next` has no tool surface — the only `tool` references are artifact tools.
- Actions reach the model as prose in the system prompt; the framework parses the returned JSON envelope.
## Evidence that this is becoming a problem
Measured against a real Loop-agent payload (an agent that iterates writing and testing SQL), replayed directly against the provider SDK outside any MJ code:
| condition | outcome |
|---|---|
| `gemini-3.7-flash`, no tools declared (what MJ sends) | **~35%** of responses returned `finishReason: MALFORMED_FUNCTION_CALL` with the generated tokens discarded |
| `gemini-3.7-flash`, one matching tool declared | **0%** malformed; the model emitted a well-formed function call **7/8** times |
| `gemini-3-flash`, `gemini-3.5-flash`, `gemini-3.6-flash` | 0/6 malformed each, same payload |
The model knows exactly which action it wants and how to call it. With no tools declared, that intent surfaces as an unparseable call and the whole response is dropped, which the driver correctly reports as "No output received from model".
Only the newest model in the family does this — so this reads as a trend rather than a one-off.
Two secondary observations:
- Loop system prompts in this workload render at **56–104KB**, much of it contract description that tool definitions would carry more efficiently (and that most providers cache better).
- `BaseAgent` deliberately records action invocations as second-person `user`-role annotations rather than assistant turns, to stop strong in-context learners imitating prose and breaking JSON parsing. Native tool calls are not prose, so that trade-off would largely dissolve.
## The catch
Naive adoption breaks the contract. In the same experiment, declaring a tool dropped *envelope* text output to **1/8** — the model called the function instead of returning `taskComplete` / `nextStep`. So this cannot be "just pass tools through"; it needs deliberate design, likely `tool_choice` control per step.
Native tool calling also only covers one branch. `LoopAgentResponse` carries `taskComplete`, `nextStep.type` (`Actions` | `Sub-Agent` | `Chat` | `Retry`), `payloadChangeRequest`, `reasoning` and `message`. Tools address `Actions`; the rest still needs structured output.
## Proposed approach
**Phase 1 — opt-in capability.** Add optional tool support to `BaseLLM` / `ChatParams`, implemented in a few drivers first (Anthropic, OpenAI, Gemini). Gate it on a `SupportsToolCalling` flag on `AIModelVendor`, following the existing `SupportsEffortLevel` / `SupportedResponseFormats` pattern. Anything without the flag keeps today's behaviour unchanged — no forced migration, and models with no tool support are unaffected.
**Phase 2 — hybrid loop.** Where the selected vendor supports tools, declare the agent's actions as tools and route `nextStep.type='Actions'` through native calls, keeping the envelope for control flow.
**Phase 3 — measure, then decide how far to take it.** Compare both paths on an agent eval suite before expanding.
## Open questions
- Is a hybrid loop (native actions + JSON control flow) coherent, or does mixing the two confuse models more than either alone?
- How much cross-provider variance is there in practice — parallel calls, forcing, whether results belong in a `tool` role or a user turn?
- Would `payloadChangeRequest` be better expressed as a tool than as envelope fields?
- How much of the observed fragility is architectural versus prompt-level? In a related fix, correcting a single malformed action exemplar in one agent prompt moved that agent's malformed rate from ~35% to ~10% (40 runs per condition, p ≈ 0.007). Auditing agent prompts for exemplars that contradict the `LoopAgentResponse` interface is far cheaper than a platform change and would help size this properly.
## Explicitly not proposed
Replacing the JSON envelope, or mandating tool calling. The vendor-agnostic text path should stay the default and remain fully supported.
Contributor guide
Research direction
Start with packages/AI/Core/src/generic/baseLLM.ts, ChatParams, the AIModelVendor capability patterns, and the Anthropic, OpenAI, and Gemini drivers named in the issue. Trace how BaseAgent builds LoopAgentResponse and dispatches Actions before evaluating the proposed hybrid flow. Done requires a documented opt-in design, provider behavior comparisons, and an agent eval plan with clear criteria for retaining the existing default path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100