Bug: tools declared by a custom_agents agent are announced but not callable by the model
- Lingua principale
- Java
- Stelle
- 10.5k
- Fork
- 1.5k
- Merge medio
- 1g 11h
- PR unite (30g)
- 127
Descrizione
# Bug: tools declared by a `custom_agents` agent are announced but not callable by the model
## Summary
When a session is created with `custom_agents=[...]` plus `agent=""`, the tools that agent
declares in its own `tools:` list are **not usable by the model**. The runtime reports them as
selected (a `subagent.selected` event lists them verbatim), but the model behaves as though it has
no tools at all: it makes **zero tool calls** and states plainly that it has no tool capable of
doing the work.
Passing the **identical** tool list via `available_tools=[...]` on the same model, same prompt,
same sandbox works correctly.
The failure is silent. There is no error, no warning, and no rejected tool call — the model simply
answers in prose. A caller that only inspects the final message (or a `subagent.selected` event)
sees a perfectly healthy-looking session.
## Environment
| | |
|---|---|
| Python SDK | `github-copilot-sdk` 1.0.11 |
| CLI / server | `copilot` 1.0.79 (also reproduced on 1.0.78) |
| Connection | `RuntimeConnection.for_uri(...)` to a `copilot --headless --server` process in a container |
| Models | reproduced with `gpt-5.3-codex` **and** `gpt-5.4` |
| `agent_mode` | `"autopilot"` |
| Permissions | `PermissionHandler.approve_all` |
| OS | server on Linux (Ubuntu 24.04 container); client on Windows |
## Reproduction
`repro.py` below runs the same request twice against the same server: once with `custom_agents`,
once with `available_tools`. Start a headless server first, then run it.
```bash
# a copilot --headless --server on :3000, published to 18097, with a /workspace/repo cwd
docker run -d --rm --name ghcprepro -p 18097:3000 \
-e COPILOT_SDK_AUTH_TOKEN="$GITHUB_TOKEN" \
-e COPILOT_CONNECTION_TOKEN="repro123" \
python repro.py gpt-5.3-codex
docker exec ghcprepro sh -c 'ls /workspace/repo/' # which file actually got created?
```
```python
"""Minimal A/B: custom_agents' declared tools vs the identical list via available_tools."""
import asyncio, sys
from copilot import CopilotClient, RuntimeConnection
from copilot.session import PermissionHandler
from copilot.session_events import SessionEventType
URI, TOKEN = "localhost:18097", "repro123"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "gpt-5.3-codex"
TOOLS = ["builtin:view", "builtin:grep", "builtin:glob",
"builtin:edit", "builtin:create", "builtin:apply_patch", "builtin:skill"]
WRITER_AGENT = {
"name": "writer",
"description": "Creates files on disk.",
"tools": TOOLS, # <-- identical list to available_tools in case B
"model": MODEL,
"prompt": "You are the Writer Agent. You create files on disk with your file tools.",
}
ASK = ("Create a file named {name} in the current directory containing exactly the text 'hello'. "
"Then tell me in one sentence which tool you used, or state plainly that you have no tool "
"capable of creating a file.")
async def run_case(name: str, *, use_custom_agent: bool) -> str:
connection = RuntimeConnection.for_uri(URI, connection_token=TOKEN)
async with CopilotClient(connection=connection, log_level="error") as client:
kwargs = {"on_permission_request": PermissionHandler.approve_all, "model": MODEL,
"streaming": True, "working_directory": "/workspace/repo"}
if use_custom_agent:
kwargs["custom_agents"] = [WRITER_AGENT]
kwargs["agent"] = "writer"
else:
kwargs["available_tools"] = TOOLS
session = await client.create_session(**kwargs)
done, text = asyncio.Event(), []
def on_event(e):
if e.type == SessionEventType.ASSISTANT_MESSAGE: text.append(e.data.content or "")
elif e.type in (SessionEventType.SESSION_IDLE, SessionEventType.SESSION_ERROR): done.set()
session.on(on_event)
await session.send(ASK.format(name=name), agent_mode="autopilot")
await asyncio.wait_for(done.wait(), timeout=120)
return f"session={session.session_id} :: {' '.join(text).strip()[:200]}"
async def main():
print("A custom_agents ->", await run_case("case_a.txt", use_custom_agent=True))
print("B available_tools->", await run_case("case_b.txt", use_custom_agent=False))
asyncio.run(main())
```
## Actual result
```
A custom_agents -> session=41113715-... :: I have no tool capable of creating a file.
B available_tools-> session=396f85a7-... :: Creating `case_b.txt` now ... I used the `apply_patch`
tool to create `case_b.txt` with exactly `hello`.
$ ls /workspace/repo/
case_b.txt # case_a.txt was never created
```
Case A reproduces on every attempt (3/3 trials, plus both models above).
### The contradiction, from the server's own session log
`~/.copilot/session-state//events.jsonl` for the **failing** case A session announces
exactly the tools that the model then cannot use:
```json
{"type":"subagent.selected","data":{"agentName":"writer","agentDisplayName":"writer",
"tools":["builtin:view","builtin:grep","builtin:glob","builtin:edit","builtin:create",
"builtin:apply_patch","builtin:skill"]}}
```
Tool invocations recorded in each session:
| session | `tool.execution_start` entries |
|---|---|
| A (custom_agents) | **none at all** |
| B (available_tools) | `apply_patch` ×1 |
So the agent's tool set is resolved and reported, but never reaches the model's callable tool set.
## Expected result
An agent's declared `tools:` should be callable by the model in that agent's turn — equivalent to
passing the same list via `available_tools`. Case A should create `case_a.txt`.
Failing that, a session whose agent declares tools the runtime cannot expose should raise an error
at `create_session`, rather than silently producing a tool-less agent.
## Impact
This is expensive to diagnose because every observable signal says the session is healthy: the
agent is selected, the declared tools are echoed back, `agent_mode` is `autopilot`, permissions are
auto-approved, and the model returns a confident, well-formed answer. Only the filesystem (or the
absence of `tool.execution_start` events) reveals that nothing happened.
In our pipeline this manifested as agents that "completed" substantial work in ~18 seconds while
writing nothing to disk, and as automated fix-up steps that never repaired anything. Because each
affected stage then failed a downstream check for an unrelated-looking reason, it took a long time
to trace back to tool availability.
## Workaround
Do not use `custom_agents` for any agent that needs tools. Pass the tool list via
`available_tools` on `create_session`, and supply the agent's instructions as a normal system
message instead of an agent definition.
## Secondary observation (possibly intended, but surprising)
`builtin:create` alone is not sufficient for file creation — with
`[view, grep, glob, edit, create]` the model reports it has no way to create a file, and creation
only succeeds once `builtin:apply_patch` is included (the model then uses `apply_patch`). If
`create` is not independently usable, it would help for its absence/aliasing to be documented, or
for `create` to be rejected as unknown rather than accepted silently.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with repro.py and run its A/B create_session cases using custom_agents versus available_tools. Trace how those options flow through create_session, then compare selected-agent tools with callable tools in session-state//events.jsonl. Done means the declared tools are callable in the custom agent turn, or create_session rejects the unsupported configuration.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- api, backend
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Attiva
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100