github / github/copilot-sdk

Bug: tools declared by a custom_agents agent are announced but not callable by the model

Đang mở
#2,356 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
Java
Star
10.5k
Fork
1.5k
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
128

Mô tả

# Bug: tools declared by a `custom_agents` agent are announced but not callable by the model

## Summary

When a session is created with `custom_agents=[...]` plus `agent=""`, the tools that agent
declares in its own `tools:` list are **not usable by the model**. The runtime reports them as
selected (a `subagent.selected` event lists them verbatim), but the model behaves as though it has
no tools at all: it makes **zero tool calls** and states plainly that it has no tool capable of
doing the work.

Passing the **identical** tool list via `available_tools=[...]` on the same model, same prompt,
same sandbox works correctly.

The failure is silent. There is no error, no warning, and no rejected tool call — the model simply
answers in prose. A caller that only inspects the final message (or a `subagent.selected` event)
sees a perfectly healthy-looking session.

## Environment

| | |
|---|---|
| Python SDK | `github-copilot-sdk` 1.0.11 |
| CLI / server | `copilot` 1.0.79 (also reproduced on 1.0.78) |
| Connection | `RuntimeConnection.for_uri(...)` to a `copilot --headless --server` process in a container |
| Models | reproduced with `gpt-5.3-codex` **and** `gpt-5.4` |
| `agent_mode` | `"autopilot"` |
| Permissions | `PermissionHandler.approve_all` |
| OS | server on Linux (Ubuntu 24.04 container); client on Windows |

## Reproduction

`repro.py` below runs the same request twice against the same server: once with `custom_agents`,
once with `available_tools`. Start a headless server first, then run it.

```bash
# a copilot --headless --server on :3000, published to 18097, with a /workspace/repo cwd
docker run -d --rm --name ghcprepro -p 18097:3000 \
-e COPILOT_SDK_AUTH_TOKEN="$GITHUB_TOKEN" \
-e COPILOT_CONNECTION_TOKEN="repro123" \

python repro.py gpt-5.3-codex
docker exec ghcprepro sh -c 'ls /workspace/repo/' # which file actually got created?
```

```python
"""Minimal A/B: custom_agents' declared tools vs the identical list via available_tools."""
import asyncio, sys
from copilot import CopilotClient, RuntimeConnection
from copilot.session import PermissionHandler
from copilot.session_events import SessionEventType

URI, TOKEN = "localhost:18097", "repro123"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "gpt-5.3-codex"
TOOLS = ["builtin:view", "builtin:grep", "builtin:glob",
"builtin:edit", "builtin:create", "builtin:apply_patch", "builtin:skill"]

WRITER_AGENT = {
"name": "writer",
"description": "Creates files on disk.",
"tools": TOOLS, # <-- identical list to available_tools in case B
"model": MODEL,
"prompt": "You are the Writer Agent. You create files on disk with your file tools.",
}

ASK = ("Create a file named {name} in the current directory containing exactly the text 'hello'. "
"Then tell me in one sentence which tool you used, or state plainly that you have no tool "
"capable of creating a file.")

async def run_case(name: str, *, use_custom_agent: bool) -> str:
connection = RuntimeConnection.for_uri(URI, connection_token=TOKEN)
async with CopilotClient(connection=connection, log_level="error") as client:
kwargs = {"on_permission_request": PermissionHandler.approve_all, "model": MODEL,
"streaming": True, "working_directory": "/workspace/repo"}
if use_custom_agent:
kwargs["custom_agents"] = [WRITER_AGENT]
kwargs["agent"] = "writer"
else:
kwargs["available_tools"] = TOOLS
session = await client.create_session(**kwargs)

done, text = asyncio.Event(), []
def on_event(e):
if e.type == SessionEventType.ASSISTANT_MESSAGE: text.append(e.data.content or "")
elif e.type in (SessionEventType.SESSION_IDLE, SessionEventType.SESSION_ERROR): done.set()
session.on(on_event)
await session.send(ASK.format(name=name), agent_mode="autopilot")
await asyncio.wait_for(done.wait(), timeout=120)
return f"session={session.session_id} :: {' '.join(text).strip()[:200]}"

async def main():
print("A custom_agents ->", await run_case("case_a.txt", use_custom_agent=True))
print("B available_tools->", await run_case("case_b.txt", use_custom_agent=False))

asyncio.run(main())
```

## Actual result

```
A custom_agents -> session=41113715-... :: I have no tool capable of creating a file.
B available_tools-> session=396f85a7-... :: Creating `case_b.txt` now ... I used the `apply_patch`
tool to create `case_b.txt` with exactly `hello`.

$ ls /workspace/repo/
case_b.txt # case_a.txt was never created
```

Case A reproduces on every attempt (3/3 trials, plus both models above).

### The contradiction, from the server's own session log

`~/.copilot/session-state//events.jsonl` for the **failing** case A session announces
exactly the tools that the model then cannot use:

```json
{"type":"subagent.selected","data":{"agentName":"writer","agentDisplayName":"writer",
"tools":["builtin:view","builtin:grep","builtin:glob","builtin:edit","builtin:create",
"builtin:apply_patch","builtin:skill"]}}
```

Tool invocations recorded in each session:

| session | `tool.execution_start` entries |
|---|---|
| A (custom_agents) | **none at all** |
| B (available_tools) | `apply_patch` ×1 |

So the agent's tool set is resolved and reported, but never reaches the model's callable tool set.

## Expected result

An agent's declared `tools:` should be callable by the model in that agent's turn — equivalent to
passing the same list via `available_tools`. Case A should create `case_a.txt`.

Failing that, a session whose agent declares tools the runtime cannot expose should raise an error
at `create_session`, rather than silently producing a tool-less agent.

## Impact

This is expensive to diagnose because every observable signal says the session is healthy: the
agent is selected, the declared tools are echoed back, `agent_mode` is `autopilot`, permissions are
auto-approved, and the model returns a confident, well-formed answer. Only the filesystem (or the
absence of `tool.execution_start` events) reveals that nothing happened.

In our pipeline this manifested as agents that "completed" substantial work in ~18 seconds while
writing nothing to disk, and as automated fix-up steps that never repaired anything. Because each
affected stage then failed a downstream check for an unrelated-looking reason, it took a long time
to trace back to tool availability.

## Workaround

Do not use `custom_agents` for any agent that needs tools. Pass the tool list via
`available_tools` on `create_session`, and supply the agent's instructions as a normal system
message instead of an agent definition.

## Secondary observation (possibly intended, but surprising)

`builtin:create` alone is not sufficient for file creation — with
`[view, grep, glob, edit, create]` the model reports it has no way to create a file, and creation
only succeeds once `builtin:apply_patch` is included (the model then uses `apply_patch`). If
`create` is not independently usable, it would help for its absence/aliasing to be documented, or
for `create` to be rejected as unknown rather than accepted silently.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu với repro.py và chạy các trường hợp A/B của create_session bằng custom_agents so với available_tools. Theo dõi cách các tùy chọn đó đi qua create_session, sau đó so sánh các công cụ của agent được chọn với các công cụ có thể gọi trong session-state//events.jsonl. Được xem là hoàn thành khi các công cụ đã khai báo có thể được gọi trong lượt của custom agent, hoặc create_session từ chối cấu hình không được hỗ trợ.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
api, backend
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
48/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.