a2ui-project / a2ui-project/a2ui

[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses

Đang mở
#2,514 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
component: agent_sdk P2 status: needs-triage type: bug
Ngôn ngữ chính
TypeScript
Star
16.4k
Fork
1.3k
Merge trung bình
2 ngày 13 giờ
Pull request đã merge (30 ngày)
134

Mô tả

# Issue Draft: LLM outputs markdown code fences around A2UI DSL when skills/instructions are retrieved via tool calls

## **Title**
`[SDK/Parser] LLM outputs markdown code fences around A2UI DSL when skills are retrieved via tool responses`

---

## **Body**

### **Context & Background**
A2UI relies on enclosing sentinel tags (e.g. `` and ``) to delimit generated user interface payloads across all inference formats (`ExpressFormat`, `ElementalFormat`, `AtomFormat`, `DirectJsonFormat`).

In traditional single-prompt agent setups, A2UI grammar rules and sentinel instructions are injected directly into the **System Prompt**. In system prompt context, LLMs treat sentinel tag rules as strict system-level output contracts and consistently emit bare `` tags.

However, modern agentic platforms (such as Anthropic API with Skills, Model Context Protocol (MCP) clients, and custom tool-retrieval agent harnesses) retrieve domain skills dynamically via tool calls (e.g. a `load_skill(name)` or `search_skills()` tool execution turn). In these architectures, the A2UI skill instructions (`SKILL.md`) are returned to the model inside a **tool response message** (`ChatMessageTool` / function result).

---

### **The Problem**
When an LLM (e.g. `google/gemini-3.5-flash`, `claude-3-5-sonnet`) receives A2UI syntax contracts and sentinel tag rules inside a **tool execution result** rather than the System Prompt, the model interprets the retrieved skill text as reference markdown documentation.

As a consequence, when generating the UI output, the LLM frequently wraps the A2UI DSL payload in markdown code fences or backticks:
```markdown
```a2ui
surface("main")
root = Column([title, form])
```
```
or:
```text
``
surface("main")
$/data = ...
``
```

Because strict sentinel tag matchers (`[\s\S]*?`) expect bare `` tags, markdown code fences cause parsing failures or `0.0` accuracy scores unless stripped.

---

### **How to Reproduce**

1. Set up an agent tool execution loop with a minimal system prompt:
```text
You are an AI assistant. You have access to a tool `load_skill(skill_name)` to retrieve specialized UI generation skills.
Available Skills: a2ui-core, a2ui-basic.
```
2. Send a user request requiring UI generation (e.g., *"Create a settings page UI on surface 'main'"*).
3. **Turn 1**: Model calls `load_skill("a2ui-core")`.
4. **Turn 2**: Tool returns the content of `a2ui-core/SKILL.md` (which includes instructions to surround output with ``).
5. **Turn 3**: Model calls `load_skill("a2ui-basic")`.
6. **Turn 4**: Tool returns the content of `a2ui-basic/SKILL.md`.
7. **Turn 5**: Model generates the UI payload.
8. **Observed Output**: The model outputs ````a2ui\nsurface("main")...\n``` ` wrapped in markdown code fences instead of bare `` tags.
9. **Expected Output**: The model should output bare `` tags without outer markdown code blocks.

---

### **Impact Analysis**
- **System Prompt vs. Tool Retrieval Discrepancy**:
- Preloaded system prompt (`skill_preloaded`): **100% Syntax Accuracy**
- Tool retrieval (`skill_interactive_tool`): **57.1% Syntax Accuracy** (due to markdown code fence wrapping)
- Tool call execution succeeds 100% of the time, but tool context shifts the model's output formatting behavior.

---

### **Proposed Mitigation / Resolutions**

1. **Parser & Payload Fixer Hardening (SDK Level)**:
Update `ExpressParser`, `ElementalParser`, `AtomParser`, `A2uiPartConverter`, and `payload_fixer.py` to automatically strip outer markdown code fences (` ```a2ui `, ` ```dsl `, ` ```xml `, ` ```json `) prior to sentinel tag extraction.
2. **Skill Prompt Instruction Tuning**:
In generated `SKILL.md` documents, explicitly add a tool-context directive:
> *"IMPORTANT: Even when this skill is loaded dynamically via a tool response, do NOT wrap your output in markdown code blocks (e.g. ```a2ui). Output bare and tags directly."*

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Look at the parsers mentioned: ExpressParser, ElementalParser, AtomParser, and the payload_fixer.py. The issue is that markdown code fences (```a2ui) are being output by LLMs when skills are retrieved via tool calls, breaking the sentinel tag matchers. Start by examining how these parsers extract content between tags. The fix likely involves modifying the parsing logic to strip outer markdown code blocks before tag matching. Check test files for parser accuracy to verify changes.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, typescript
Lĩnh vực
tooling
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
65/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.