MemberJunction / MemberJunction/MJ
Data Analysis skill requires entity discovery but grants no way to do it — silently burns agent iterations
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
**Summary**
The `Data Analysis & Queries` skill instructs agents to explore the schema and to call `Get Entity Details` with "the **exact entity name**", but grants no action capable of discovering an entity name, and never states that MJ entity names carry an `MJ: ` prefix. Agents are therefore forced to guess, and every wrong guess costs an iteration.
On `Codesmith Agent` — the only skill-accepting agent with an iteration budget — this is fatal. The run dies with `Maximum iteration limit of 10 exceeded`, a message that names the budget and hides the cause.
**Evidence**
Two `Codesmith Agent` runs in a dev database failed identically, 76 minutes apart, on the same wrong name. They are the only two runs in that database that ever hit the iteration limit.
| Started | Result | First failing step |
|---|---|---|
| `2026-09-16T01:28:54` | 10 iterations, limit exceeded | `Get Entity Details` → `ENTITY_NOT_FOUND` |
| `2026-09-16T02:44:28` | 10 iterations, limit exceeded | `Get Entity Details` → `ENTITY_NOT_FOUND` |
The two distinct failures, from `AIAgentRunStep` rows:
```
ENTITY_NOT_FOUND: Entity 'AI Models' not found. Did you mean: 'MJ: AI Models'?
Entity names must match exactly, including any 'MJ: ' prefix.
DANGEROUS_QUERY: Access to database system catalogs / metadata objects is not allowed
Query: SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.VIEWS WHERE ...
```
The second is the agent falling back to the only schema-inspection tool it has — `Run Ad-hoc Query` — and hitting `SQLExpressionValidator`.
In the second run, 2 of 10 iterations were spent recovering from these two avoidable failures. The remaining steps all succeeded, so the agent was doing legitimate work with a 20% smaller budget than it appeared to have.
**Root cause**
The skill grants five actions:
`Aggregate Data`, `Get Entity Details`, `Run Ad-hoc Query`, `Run Stored Query`, `Search Query Catalog`
Two actions exist for exactly this problem and neither is granted:
| Action | Its own doc comment |
|---|---|
| `Get Entity List` | "Discover available entities **without guessing names** … Identify correct entity names before detailed exploration" |
| `Explore Database Schema` | "…using MemberJunction's rich metadata layer **instead of raw `INFORMATION_SCHEMA` queries**" |
Across the entire skill library, **zero** skills grant either action, and the only skill that grants `Get Entity Details` — which requires an exact name — is this one.
The escape hatch has the same hole: the skill says to delegate complex work to the `Query Strategist` sub-agent, which has four actions and also lacks both.
Nothing teaches the naming rule either. `metadata/ai-skills/templates/data-analysis-queries.skill.md` and `metadata/prompts/templates/agents/codesmith.template.md` each contain **zero** occurrences of `MJ: `.
This is already a known agent behaviour. From `packages/Actions/CoreActions/src/custom/data/get-entity-details.action.ts`:
> Agents **commonly** pass display-style names (e.g., 'AI Agents') when the real entity name carries an 'MJ: ' prefix … pointing at the exact candidates lets them recover in one step instead of guessing.
The mitigation chosen was a better error message. That costs one iteration per occurrence rather than preventing the mistake.
**Not a regression**
Neither action has ever been wired to any skill:
```
git log -S "Get Entity List" -- metadata/ai-skills/ → no commits
git log -S "Explore Database Schema" -- metadata/ai-skills/ → no commits
```
`data-analysis-queries.skill.md` has exactly one commit in its history: the one that created it.
Both actions predate the skill by nearly nine months. `get-entity-list.action.ts` and `get-entity-details.action.ts` were added in the **same commit** on 2025-10-15; the skill was authored 2026-07-01. The skill took one and left its companion behind.
**Origin**
- #3013 (merged 2026-07-01 23:29) created the skill. Its commit message lists the five actions chosen; `Get Entity List` is absent.
- #3014 (merged 2026-07-01 23:34) set `AcceptsSkills='All'` on six top-level agents, which is how the skill reaches `Codesmith Agent` — there is no explicit link between them.
#3013 is the one to fix. #3014 only made the fault reachable.
**Why only Codesmith fails**
Six agents accept all skills. Only one has an iteration budget:
| Agent | `MaxIterationsPerRun` |
|---|---|
| **Codesmith Agent** | **10** |
| Query Builder | none |
| Sage | none |
| Marketing Agent | none |
| Knowledge Agent | none |
| Research Agent | none |
The other five waste the same two iterations with no ceiling to hit, so the waste is silent. Codesmith's limit of 10 was set on 2025-10-19 for a code-execution agent, eight months before skills existed, and was never revisited when #3014 attached a schema-exploration skill to it.
**Suggested fix**
Metadata only — no code change, no migration.
1. Grant `Get Entity List` and `Explore Database Schema` to the skill in `metadata/ai-skills/.core-skills.json`, and to the `Query Strategist` sub-agent.
2. In `data-analysis-queries.skill.md`, state the `MJ: ` prefix rule and direct the agent to call `Get Entity List` before `Get Entity Details`. The current text says "use the **exact entity name**" without saying what one looks like or how to obtain it.
3. Review whether 10 iterations is right for `Codesmith Agent` now that it auto-accepts schema-exploration skills.
4. Consider auditing other skills for instructions that require ungranted capabilities — this class of fault is silent until an agent with a budget hits it.
**Note on diagnosis**
The reported symptom (`Maximum iteration limit of 10 exceeded`) points at the budget, not the cause, and the two underlying failures are only visible in `AIAgentRunStep` rows. Anything that consumes iterations invisibly will present this way. A run-level summary of failed steps in the error message would have made this self-diagnosing.
Contributor guide
Research direction
Start with metadata/ai-skills/.core-skills.json and metadata/ai-skills/templates/data-analysis-queries.skill.md, then inspect the Query Strategist configuration and metadata/prompts/templates/agents/codesmith.template.md. Verify the relevant actions are granted, the MJ: prefix and discovery flow are documented, and the iteration-budget question is reviewed without making code or migration changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json, markdown, sql
- Domain
- ai, documentation
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100