a2ui-project / a2ui-project/a2ui

[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses

Aperta
#2,514 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
component: agent_sdk P2 status: needs-triage type: bug
Lingua principale
TypeScript
Stelle
16.4k
Fork
1.3k
Merge medio
2g 13h
PR unite (30g)
134

Descrizione

# Issue Draft: LLM outputs markdown code fences around A2UI DSL when skills/instructions are retrieved via tool calls

## **Title**
`[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses`

---

## **Body**

### **Context & Background**
A2UI relies on enclosing sentinel tags (e.g. `` and ``) to delimit generated user interface payloads across all inference formats (`ExpressFormat`, `ElementalFormat`, `AtomFormat`, `DirectJsonFormat`).

In traditional single-prompt agent setups, A2UI grammar rules and sentinel instructions are injected directly into the **System Prompt**. In system prompt context, LLMs treat sentinel tag rules as strict system-level output contracts and consistently emit bare `` tags.

However, modern agentic platforms (such as Anthropic API with Skills, Model Context Protocol (MCP) clients, and custom tool-retrieval agent harnesses) retrieve domain skills dynamically via tool calls (e.g. a `load_skill(name)` or `search_skills()` tool execution turn). In these architectures, the A2UI skill instructions (`SKILL.md`) are returned to the model inside a **tool response message** (`ChatMessageTool` / function result).

---

### **The Problem**
When an LLM (e.g. `google/gemini-3.5-flash`, `claude-3-5-sonnet`) receives A2UI syntax contracts and sentinel tag rules inside a **tool execution result** rather than the System Prompt, the model interprets the retrieved skill text as reference markdown documentation.

As a consequence, when generating the UI output, the LLM frequently wraps the A2UI DSL payload in markdown code fences or backticks:
```markdown
```a2ui
surface("main")
root = Column([title, form])
```
```
or:
```text
``
surface("main")
$/data = ...
``
```

Because strict sentinel tag matchers (`[\s\S]*?`) expect bare `` tags, markdown code fences cause parsing failures or `0.0` accuracy scores unless stripped.

---

### **How to Reproduce**

1. Set up an agent tool execution loop with a minimal system prompt:
```text
You are an AI assistant. You have access to a tool `load_skill(skill_name)` to retrieve specialized UI generation skills.
Available Skills: a2ui-core, a2ui-basic.
```
2. Send a user request requiring UI generation (e.g., *"Create a settings page UI on surface 'main'"*).
3. **Turn 1**: Model calls `load_skill("a2ui-core")`.
4. **Turn 2**: Tool returns the content of `a2ui-core/SKILL.md` (which includes instructions to surround output with ``).
5. **Turn 3**: Model calls `load_skill("a2ui-basic")`.
6. **Turn 4**: Tool returns the content of `a2ui-basic/SKILL.md`.
7. **Turn 5**: Model generates the UI payload.
8. **Observed Output**: The model outputs ````a2ui\nsurface("main")...\n``` ` wrapped in markdown code fences instead of bare `` tags.
9. **Expected Output**: The model should output bare `` tags without outer markdown code blocks.

---

### **Impact Analysis**
- **System Prompt vs. Tool Retrieval Discrepancy**:
- Preloaded system prompt (`skill_preloaded`): **100% Syntax Accuracy**
- Tool retrieval (`skill_interactive_tool`): **57.1% Syntax Accuracy** (due to markdown code fence wrapping)
- Tool call execution succeeds 100% of the time, but tool context shifts the model's output formatting behavior.

---

### **Proposed Mitigation / Resolutions**

1. **Parser & Payload Fixer Hardening (SDK Level)**:
Update `ExpressParser`, `ElementalParser`, `AtomParser`, `A2uiPartConverter`, and `payload_fixer.py` to automatically strip outer markdown code fences (` ```a2ui `, ` ```dsl `, ` ```xml `, ` ```json `) prior to sentinel tag extraction.
2. **Skill Prompt Instruction Tuning**:
In generated `SKILL.md` documents, explicitly add a tool-context directive:
> *"IMPORTANT: Even when this skill is loaded dynamically via a tool response, do NOT wrap your output in markdown code blocks (e.g. ```a2ui). Output bare and tags directly."*

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.