MemberJunction / MemberJunction/MJ

Data Analysis skill requires entity discovery but grants no way to do it — silently burns agent iterations

Open
#4,528 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

**Summary**

The `Data Analysis & Queries` skill instructs agents to explore the schema and to call `Get Entity Details` with "the **exact entity name**", but grants no action capable of discovering an entity name, and never states that MJ entity names carry an `MJ: ` prefix. Agents are therefore forced to guess, and every wrong guess costs an iteration.

On `Codesmith Agent` — the only skill-accepting agent with an iteration budget — this is fatal. The run dies with `Maximum iteration limit of 10 exceeded`, a message that names the budget and hides the cause.

**Evidence**

Two `Codesmith Agent` runs in a dev database failed identically, 76 minutes apart, on the same wrong name. They are the only two runs in that database that ever hit the iteration limit.

| Started | Result | First failing step |
|---|---|---|
| `2026-09-16T01:28:54` | 10 iterations, limit exceeded | `Get Entity Details` → `ENTITY_NOT_FOUND` |
| `2026-09-16T02:44:28` | 10 iterations, limit exceeded | `Get Entity Details` → `ENTITY_NOT_FOUND` |

The two distinct failures, from `AIAgentRunStep` rows:

```
ENTITY_NOT_FOUND: Entity 'AI Models' not found. Did you mean: 'MJ: AI Models'?
Entity names must match exactly, including any 'MJ: ' prefix.

DANGEROUS_QUERY: Access to database system catalogs / metadata objects is not allowed
Query: SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.VIEWS WHERE ...
```

The second is the agent falling back to the only schema-inspection tool it has — `Run Ad-hoc Query` — and hitting `SQLExpressionValidator`.

In the second run, 2 of 10 iterations were spent recovering from these two avoidable failures. The remaining steps all succeeded, so the agent was doing legitimate work with a 20% smaller budget than it appeared to have.

**Root cause**

The skill grants five actions:

`Aggregate Data`, `Get Entity Details`, `Run Ad-hoc Query`, `Run Stored Query`, `Search Query Catalog`

Two actions exist for exactly this problem and neither is granted:

| Action | Its own doc comment |
|---|---|
| `Get Entity List` | "Discover available entities **without guessing names** … Identify correct entity names before detailed exploration" |
| `Explore Database Schema` | "…using MemberJunction's rich metadata layer **instead of raw `INFORMATION_SCHEMA` queries**" |

Across the entire skill library, **zero** skills grant either action, and the only skill that grants `Get Entity Details` — which requires an exact name — is this one.

The escape hatch has the same hole: the skill says to delegate complex work to the `Query Strategist` sub-agent, which has four actions and also lacks both.

Nothing teaches the naming rule either. `metadata/ai-skills/templates/data-analysis-queries.skill.md` and `metadata/prompts/templates/agents/codesmith.template.md` each contain **zero** occurrences of `MJ: `.

This is already a known agent behaviour. From `packages/Actions/CoreActions/src/custom/data/get-entity-details.action.ts`:

> Agents **commonly** pass display-style names (e.g., 'AI Agents') when the real entity name carries an 'MJ: ' prefix … pointing at the exact candidates lets them recover in one step instead of guessing.

The mitigation chosen was a better error message. That costs one iteration per occurrence rather than preventing the mistake.

**Not a regression**

Neither action has ever been wired to any skill:

```
git log -S "Get Entity List" -- metadata/ai-skills/ → no commits
git log -S "Explore Database Schema" -- metadata/ai-skills/ → no commits
```

`data-analysis-queries.skill.md` has exactly one commit in its history: the one that created it.

Both actions predate the skill by nearly nine months. `get-entity-list.action.ts` and `get-entity-details.action.ts` were added in the **same commit** on 2025-10-15; the skill was authored 2026-07-01. The skill took one and left its companion behind.

**Origin**

- #3013 (merged 2026-07-01 23:29) created the skill. Its commit message lists the five actions chosen; `Get Entity List` is absent.
- #3014 (merged 2026-07-01 23:34) set `AcceptsSkills='All'` on six top-level agents, which is how the skill reaches `Codesmith Agent` — there is no explicit link between them.

#3013 is the one to fix. #3014 only made the fault reachable.

**Why only Codesmith fails**

Six agents accept all skills. Only one has an iteration budget:

| Agent | `MaxIterationsPerRun` |
|---|---|
| **Codesmith Agent** | **10** |
| Query Builder | none |
| Sage | none |
| Marketing Agent | none |
| Knowledge Agent | none |
| Research Agent | none |

The other five waste the same two iterations with no ceiling to hit, so the waste is silent. Codesmith's limit of 10 was set on 2025-10-19 for a code-execution agent, eight months before skills existed, and was never revisited when #3014 attached a schema-exploration skill to it.

**Suggested fix**

Metadata only — no code change, no migration.

1. Grant `Get Entity List` and `Explore Database Schema` to the skill in `metadata/ai-skills/.core-skills.json`, and to the `Query Strategist` sub-agent.
2. In `data-analysis-queries.skill.md`, state the `MJ: ` prefix rule and direct the agent to call `Get Entity List` before `Get Entity Details`. The current text says "use the **exact entity name**" without saying what one looks like or how to obtain it.
3. Review whether 10 iterations is right for `Codesmith Agent` now that it auto-accepts schema-exploration skills.
4. Consider auditing other skills for instructions that require ungranted capabilities — this class of fault is silent until an agent with a budget hits it.

**Note on diagnosis**

The reported symptom (`Maximum iteration limit of 10 exceeded`) points at the budget, not the cause, and the two underlying failures are only visible in `AIAgentRunStep` rows. Anything that consumes iterations invisibly will present this way. A run-level summary of failed steps in the error message would have made this self-diagnosing.

Contributor guide

Open the contributing guide

Research direction

Start with metadata/ai-skills/.core-skills.json and metadata/ai-skills/templates/data-analysis-queries.skill.md, then inspect the Query Strategist configuration and metadata/prompts/templates/agents/codesmith.template.md. Verify the relevant actions are granted, the MJ: prefix and discovery flow are documented, and the iteration-budget question is reviewed without making code or migration changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, markdown, sql
Domain
ai, documentation
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.