a2ui-project / a2ui-project/a2ui
[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses
- Langage dominant
- TypeScript
- Étoiles
- 16.4k
- Forks
- 1.3k
- Merge moyen
- 2 j 13 h
- PR mergées (30 j)
- 134
Description
# Issue Draft: LLM outputs markdown code fences around A2UI DSL when skills/instructions are retrieved via tool calls
## **Title**
`[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses`
---
## **Body**
### **Context & Background**
A2UI relies on enclosing sentinel tags (e.g. `` and ``) to delimit generated user interface payloads across all inference formats (`ExpressFormat`, `ElementalFormat`, `AtomFormat`, `DirectJsonFormat`).
In traditional single-prompt agent setups, A2UI grammar rules and sentinel instructions are injected directly into the **System Prompt**. In system prompt context, LLMs treat sentinel tag rules as strict system-level output contracts and consistently emit bare `` tags.
However, modern agentic platforms (such as Anthropic API with Skills, Model Context Protocol (MCP) clients, and custom tool-retrieval agent harnesses) retrieve domain skills dynamically via tool calls (e.g. a `load_skill(name)` or `search_skills()` tool execution turn). In these architectures, the A2UI skill instructions (`SKILL.md`) are returned to the model inside a **tool response message** (`ChatMessageTool` / function result).
---
### **The Problem**
When an LLM (e.g. `google/gemini-3.5-flash`, `claude-3-5-sonnet`) receives A2UI syntax contracts and sentinel tag rules inside a **tool execution result** rather than the System Prompt, the model interprets the retrieved skill text as reference markdown documentation.
As a consequence, when generating the UI output, the LLM frequently wraps the A2UI DSL payload in markdown code fences or backticks:
```markdown
```a2ui
surface("main")
root = Column([title, form])
```
```
or:
```text
``
surface("main")
$/data = ...
``
```
Because strict sentinel tag matchers (`[\s\S]*?`) expect bare `` tags, markdown code fences cause parsing failures or `0.0` accuracy scores unless stripped.
---
### **How to Reproduce**
1. Set up an agent tool execution loop with a minimal system prompt:
```text
You are an AI assistant. You have access to a tool `load_skill(skill_name)` to retrieve specialized UI generation skills.
Available Skills: a2ui-core, a2ui-basic.
```
2. Send a user request requiring UI generation (e.g., *"Create a settings page UI on surface 'main'"*).
3. **Turn 1**: Model calls `load_skill("a2ui-core")`.
4. **Turn 2**: Tool returns the content of `a2ui-core/SKILL.md` (which includes instructions to surround output with ``).
5. **Turn 3**: Model calls `load_skill("a2ui-basic")`.
6. **Turn 4**: Tool returns the content of `a2ui-basic/SKILL.md`.
7. **Turn 5**: Model generates the UI payload.
8. **Observed Output**: The model outputs ````a2ui\nsurface("main")...\n``` ` wrapped in markdown code fences instead of bare `` tags.
9. **Expected Output**: The model should output bare `` tags without outer markdown code blocks.
---
### **Impact Analysis**
- **System Prompt vs. Tool Retrieval Discrepancy**:
- Preloaded system prompt (`skill_preloaded`): **100% Syntax Accuracy**
- Tool retrieval (`skill_interactive_tool`): **57.1% Syntax Accuracy** (due to markdown code fence wrapping)
- Tool call execution succeeds 100% of the time, but tool context shifts the model's output formatting behavior.
---
### **Proposed Mitigation / Resolutions**
1. **Parser & Payload Fixer Hardening (SDK Level)**:
Update `ExpressParser`, `ElementalParser`, `AtomParser`, `A2uiPartConverter`, and `payload_fixer.py` to automatically strip outer markdown code fences (` ```a2ui `, ` ```dsl `, ` ```xml `, ` ```json `) prior to sentinel tag extraction.
2. **Skill Prompt Instruction Tuning**:
In generated `SKILL.md` documents, explicitly add a tool-context directive:
> *"IMPORTANT: Even when this skill is loaded dynamically via a tool response, do NOT wrap your output in markdown code blocks (e.g. ```a2ui). Output bare and tags directly."*
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.