a2ui-project / a2ui-project/a2ui

[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses

Open
#2,514 2 comments 0 reactions 0 assignees View on GitHub
component: agent_sdk P2 status: needs-triage type: bug
Dominant language
TypeScript
Stars
16.4k
Forks
1.3k
Avg merge
2d 13h
Merged PRs (30d)
134

Description

# Issue Draft: LLM outputs markdown code fences around A2UI DSL when skills/instructions are retrieved via tool calls

## **Title**
`[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses`

---

## **Body**

### **Context & Background**
A2UI relies on enclosing sentinel tags (e.g. `` and ``) to delimit generated user interface payloads across all inference formats (`ExpressFormat`, `ElementalFormat`, `AtomFormat`, `DirectJsonFormat`).

In traditional single-prompt agent setups, A2UI grammar rules and sentinel instructions are injected directly into the **System Prompt**. In system prompt context, LLMs treat sentinel tag rules as strict system-level output contracts and consistently emit bare `` tags.

However, modern agentic platforms (such as Anthropic API with Skills, Model Context Protocol (MCP) clients, and custom tool-retrieval agent harnesses) retrieve domain skills dynamically via tool calls (e.g. a `load_skill(name)` or `search_skills()` tool execution turn). In these architectures, the A2UI skill instructions (`SKILL.md`) are returned to the model inside a **tool response message** (`ChatMessageTool` / function result).

---

### **The Problem**
When an LLM (e.g. `google/gemini-3.5-flash`, `claude-3-5-sonnet`) receives A2UI syntax contracts and sentinel tag rules inside a **tool execution result** rather than the System Prompt, the model interprets the retrieved skill text as reference markdown documentation.

As a consequence, when generating the UI output, the LLM frequently wraps the A2UI DSL payload in markdown code fences or backticks:
```markdown
```a2ui
surface("main")
root = Column([title, form])
```
```
or:
```text
``
surface("main")
$/data = ...
``
```

Because strict sentinel tag matchers (`[\s\S]*?`) expect bare `` tags, markdown code fences cause parsing failures or `0.0` accuracy scores unless stripped.

---

### **How to Reproduce**

1. Set up an agent tool execution loop with a minimal system prompt:
```text
You are an AI assistant. You have access to a tool `load_skill(skill_name)` to retrieve specialized UI generation skills.
Available Skills: a2ui-core, a2ui-basic.
```
2. Send a user request requiring UI generation (e.g., *"Create a settings page UI on surface 'main'"*).
3. **Turn 1**: Model calls `load_skill("a2ui-core")`.
4. **Turn 2**: Tool returns the content of `a2ui-core/SKILL.md` (which includes instructions to surround output with ``).
5. **Turn 3**: Model calls `load_skill("a2ui-basic")`.
6. **Turn 4**: Tool returns the content of `a2ui-basic/SKILL.md`.
7. **Turn 5**: Model generates the UI payload.
8. **Observed Output**: The model outputs ````a2ui\nsurface("main")...\n``` ` wrapped in markdown code fences instead of bare `` tags.
9. **Expected Output**: The model should output bare `` tags without outer markdown code blocks.

---

### **Impact Analysis**
- **System Prompt vs. Tool Retrieval Discrepancy**:
- Preloaded system prompt (`skill_preloaded`): **100% Syntax Accuracy**
- Tool retrieval (`skill_interactive_tool`): **57.1% Syntax Accuracy** (due to markdown code fence wrapping)
- Tool call execution succeeds 100% of the time, but tool context shifts the model's output formatting behavior.

---

### **Proposed Mitigation / Resolutions**

1. **Parser & Payload Fixer Hardening (SDK Level)**:
Update `ExpressParser`, `ElementalParser`, `AtomParser`, `A2uiPartConverter`, and `payload_fixer.py` to automatically strip outer markdown code fences (` ```a2ui `, ` ```dsl `, ` ```xml `, ` ```json `) prior to sentinel tag extraction.
2. **Skill Prompt Instruction Tuning**:
In generated `SKILL.md` documents, explicitly add a tool-context directive:
> *"IMPORTANT: Even when this skill is loaded dynamically via a tool response, do NOT wrap your output in markdown code blocks (e.g. ```a2ui). Output bare and tags directly."*

Contributor guide

Open the contributing guide

Research direction

Look at the parsers mentioned: ExpressParser, ElementalParser, AtomParser, and the payload_fixer.py. The issue is that markdown code fences (```a2ui) are being output by LLMs when skills are retrieved via tool calls, breaking the sentinel tag matchers. Start by examining how these parsers extract content between tags. The fix likely involves modifying the parsing logic to strip outer markdown code blocks before tag matching. Check test files for parser accuracy to verify changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.