a2ui-project / a2ui-project/a2ui

generate_system_prompt() conflicts with SendA2uiToClientToolset — injects <a2ui-json> tag instructions causing non-deterministic tool use

Đang mở
#1,045 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
component: agent_sdk P2 type: bug type: feature/enhancement
Ngôn ngữ chính
TypeScript
Star
16.4k
Fork
1.3k
Merge trung bình
2 ngày 13 giờ
Pull request đã merge (30 ngày)
134

Mô tả

**Severity:** Medium — affects all users of tool-based A2UI emission

---

## Summary

`A2uiSchemaManager.generate_system_prompt()` unconditionally injects `` tag workflow instructions into the system prompt, even when `include_schema=False`. When used alongside `SendA2uiToClientToolset` (the tool-based approach), this creates conflicting instructions:

- The system prompt tells the LLM: *"wrap A2UI JSON in `` tags"* (inline text output)
- The toolset tells the LLM: *"call `send_a2ui_json_to_client`"* (structured function call)

With Gemini 2.5 Flash, this causes **non-deterministic behavior** — the model randomly chooses between proper tool use, malformed code-style output, or inline text with tags.

## Reproduction

### Setup

```python
# agent.py
from google.adk.agents.llm_agent import LlmAgent
from a2ui.core.schema.manager import A2uiSchemaManager
from a2ui.basic_catalog.provider import BasicCatalog
from a2ui.core.schema.constants import VERSION_0_8
from a2ui.adk.a2a_extension.send_a2ui_to_client_toolset import SendA2uiToClientToolset

schema_manager = A2uiSchemaManager(
version=VERSION_0_8,
catalogs=[BasicCatalog.get_config(
version=VERSION_0_8,
examples_path="examples/standard",
)],
)
selected_catalog = schema_manager.get_selected_catalog()
examples = schema_manager.load_examples(selected_catalog, validate=False)

# This injects tag instructions — even with include_schema=False
instruction = schema_manager.generate_system_prompt(
role_description="You are a helpful assistant.",
include_schema=False,
include_examples=False,
)

root_agent = LlmAgent(
model="gemini-2.5-flash",
name="my_agent",
instruction=instruction,
tools=[
SendA2uiToClientToolset(
a2ui_enabled=True,
a2ui_catalog=selected_catalog,
a2ui_examples=examples or "",
),
],
)
```

Run with `adk api_server` and send via `/run_sse` with `streaming: true`.

### What gets injected

Even with `include_schema=False, include_examples=False`, `generate_system_prompt()` always adds:

```
The generated response MUST follow these rules:
- The response can contain one or more A2UI JSON blocks.
- Each A2UI JSON block MUST be wrapped in `` and `` tags.
- Between or around these blocks, you can provide conversational text.
- The JSON part MUST be a single, raw JSON object...
```

This directly conflicts with `SendA2uiToClientToolset`, which registers `send_a2ui_json_to_client` as a structured function call.

### Observed behavior

Sending "Create a contact card for Alice, email alice@test.com" produces three different outcomes non-deterministically:

**1. Proper tool call (desired)** — ~33% of the time with `gemini-2.5-flash`:
```
functionCall: { name: "send_a2ui_json_to_client", args: { a2ui_json: "[...]" } }
```

**2. Malformed code-style call** — ~20%:
```
Malformed function call: print(default_api.send_a2ui_json_to_client(a2ui_json='[...]'))
```

**3. Inline text with `` tags** — ~40%:
```
Here is the contact card:

[{"beginRendering": ...}, {"surfaceUpdate": ...}]

```

### Test results

We ran systematic tests across models and `FunctionCallingConfigMode`:

| Model | FunctionCallingConfigMode | Success Rate | Failures |
|-------|--------------------------|-------------|----------|
| gemini-2.5-flash | AUTO (default) | 1/3 (33%) | 1 malformed, 1 inline |
| gemini-2.5-flash | ANY | 3/3 (100%) | — |
| gemini-2.5-pro | AUTO (default) | 2/3 (67%) | 1 unknown |
| gemini-2.5-pro | ANY | 3/3 (100%) | — |

## Current workarounds

### Workaround 1: Skip `generate_system_prompt` entirely

Write the instruction directly. `SendA2uiToClientToolset.process_llm_request()` injects schema and examples automatically — `generate_system_prompt` is redundant when using the toolset.

```python
# Instead of generate_system_prompt(), write directly:
instruction = """You are a helpful assistant.
When visual presentation helps, call the send_a2ui_json_to_client tool.
For simple questions, respond with text only."""

# Result: 15/15 success with gemini-2.5-flash + streaming=true
```

### Workaround 2: Force `FunctionCallingConfigMode.ANY`

```python
from google.genai import types

root_agent = LlmAgent(
model="gemini-2.5-flash",
instruction=instruction,
generate_content_config=types.GenerateContentConfig(
tool_config=types.ToolConfig(
function_calling_config=types.FunctionCallingConfig(
mode=types.FunctionCallingConfigMode.ANY,
)
)
),
tools=[SendA2uiToClientToolset(...)],
)
# Result: 3/3 success even with the large 52k prompt
```

**Drawback of `ANY` mode:** The model *must* call a tool on every turn. Simple text queries like "What is 2+2?" will still trigger a `send_a2ui_json_to_client` call instead of responding with plain text. This breaks the natural mixed-mode behavior where the agent decides whether UI is appropriate.

## Suggested fix

### Option A: Add `include_workflow` parameter

```python
instruction = schema_manager.generate_system_prompt(
role_description="You are a helpful assistant.",
include_schema=False,
include_examples=False,
include_workflow=False, # NEW: skip tag instructions
)
```

### Option B: Auto-detect tool-based usage

If `SendA2uiToClientToolset` is registered, `generate_system_prompt` should not inject the `` tag workflow since the toolset handles prompt injection via `process_llm_request`.

### Option C: Document incompatibility

At minimum, document that `generate_system_prompt()` should NOT be used with `SendA2uiToClientToolset`. The rizzcharts sample works around this by using a strong `WORKFLOW_DESCRIPTION` that overrides the injected tag instructions, but this is fragile and non-obvious.

## Note on the rizzcharts sample

The `rizzcharts` sample in this repo uses both `generate_system_prompt()` and `SendA2uiToClientToolset` together. It works because:

1. Its `WORKFLOW_DESCRIPTION` explicitly says "Call the `send_a2ui_json_to_client` tool" — overriding the tag instructions
2. It uses `BuiltInPlanner` with `ThinkingConfig(include_thoughts=True)` which may improve function call reliability
3. It may be tested with models or configurations that handle the conflict better

However, following the same pattern without the strong workflow override produces the non-deterministic behavior described above.

## Environment

- `google-adk==1.28.0`
- `a2ui-agent==0.1.0` (from source, `agent_sdks/python`)
- `gemini-2.5-flash` and `gemini-2.5-pro` via Vertex AI
- Python 3.14, macOS

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Look at a2ui/core/schema/manager.py for the generate_system_prompt method and a2ui/adk/a2a_extension/send_a2ui_to_client_toolset.py for the toolset. The conflict is in the injected instructions. Test with the provided reproduction script to see the non-deterministic behavior. The fix likely involves modifying generate_system_prompt to conditionally skip the tag instructions when used with the toolset.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, typescript
Lĩnh vực
backend-api-design, tooling
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
45/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.