Ask: Fall back to the parent model when a sub-agent's model fails at call-time
- Langage dominant
- Java
- Étoiles
- 10.5k
- Forks
- 1.5k
- Merge moyen
- 1 j 11 h
- PR mergées (30 j)
- 128
Description
**Component:** `@github/copilot-sdk` + `copilot-agent-runtime` — sub-agent model resolution
**Status:** Behavior does not match the documented `CustomAgentConfig.model` contract
> Model IDs below are redacted placeholders: **``** = a model in the catalog but not onboarded / no-capacity for the test user; **``** = the (available) parent session model.
---
## 1. TL;DR
**Ask.** When a sub-agent's configured model fails at **call-time** — i.e. when the runtime issues the actual model request to CAPI (the Copilot model API) — with a model-unavailable `400` (unknown / not onboarded / no capacity), the runtime should **fall back to the parent session model and retry that sub-agent's call** — instead of letting the sub-agent's model call fail.
**Why this is the contract, not a feature request.** The SDK already documents this fallback: `CustomAgentConfig.model` *"falling back to the parent session model if unavailable"* (`types.d.ts:1160-1165`). We observe it does **not** happen at call-time.
**One-line evidence.** A sub-agent set to `` under a `` parent → the model is sent to CAPI unchanged → CAPI returns `400 Unknown model ""` → no retry with the parent model → the sub-agent's call fails.
---
## 2. The documented contract
A host application embeds `@github/copilot-sdk` and dispatches sub-agents via `SessionConfig.customAgents: CustomAgentConfig[]`. Each sub-agent's model is a string on `CustomAgentConfig.model`, documented as:
```ts
// @github/copilot-sdk/dist/types.d.ts:1160-1165
/**
* Model identifier for this agent (e.g. "claude-haiku-4.5").
* When set, the runtime will attempt to use this model for the agent,
* falling back to the parent session model if unavailable.
*/
model?: string;
```
Three cases follow; we rely on the third:
| `model` value | Documented behavior |
|---|---|
| set to a valid model | used for the sub-agent |
| omitted / `undefined` | sub-agent inherits the parent session model |
| **set but unavailable** | **fall back to the parent session model and continue** |
The third row is what fails at call-time.
---
## 3. The gap (observed)
For a sub-agent model that is unavailable through the user's CAPI endpoint — e.g. not onboarded or no capacity (`` in our repro) — we observe in the outgoing request and in the event stream:
1. The configured model is sent to CAPI **unchanged** (logged: the sub-agent's first request carries ``).
2. CAPI returns **`400`**: `Unknown model "". Please ensure the model is onboarded.`
3. The runtime emits an **ephemeral, telemetry-only** `model.call_failure` (`source=subagent`, `statusCode=400`, `model=""`).
4. **No fallback:** no call is re-issued with the parent model. The sub-agent's model call fails.
> **Confirmed gap:** the documented *"fall back to the parent session model if unavailable"* is not honored for a call-time `400`. (Our source-level reading of *why* is in Appendix §6.3 — hypothesis only.)
---
## 4. Evidence
### 4.1 Setup
- A host embedding `@github/copilot-sdk`; a local end-to-end run.
- Parent session model: ``.
- Three sub-agents dispatched via `SessionConfig.customAgents`:
- **A**, **C** → `` (unavailable → 400)
- **B** → `` (available → success)
- All sub-agent event hooks, `model.call_failure`, and the model sent on each request are instrumented.
### 4.2 Event sequence
1. `subagent.started` ×3 (A, B, C).
2. First model request per sub-agent: A→``, B→``, C→``.
3. `model.call_failure` ×2 (A, C) — ephemeral / telemetry-only:
```json
{
"source": "subagent",
"initiator": "sub-agent",
"statusCode": 400,
"model": "",
"errorMessage": "...Unknown model \"\". Please ensure the model is onboarded.",
"agentId": "call_xxx"
}
```
(`agentId` is the Task tool-call ID, not the sub-agent name.)
4. No parent-model retry follows. The sub-agent's call fails.
### 4.3 Downstream behavior after the 400 — not confirmed
Our probe covered only background/detached dispatch. We did not verify what happens after the `400` on the normal foreground path, so we make **no claim** about whether the failure stays isolated to the sub-agent or affects the parent session. Appendix §6.3 records a source-level hypothesis only.
### 4.4 Why the host cannot recover this itself
- `model.call_failure` is `ephemeral` telemetry (`session-events.d.ts:2821-2887`, `ephemeral: true`); we found **no in-turn hook** that lets the host intercept the `400` and retry the sub-agent with another model within the same turn.
- Per the SDK types, `customAgents` is applied when a session is created or resumed (`types.d.ts` — `SessionConfigBase.customAgents`, inherited by `ResumeSessionConfig`). The host therefore cannot change the model during the failing turn — only after resuming the session, which is too late to recover the first turn. So the fallback must occur inside the runtime.
---
## 5. What we're asking
**1. Honor the documented fallback at call-time.**
When CAPI rejects a sub-agent's `CustomAgentConfig.model` as unavailable (`400`), retry that sub-agent's call once with the parent session model, then continue the turn — matching the `model?` contract.
The runtime can identify this condition from the failure data already present: `statusCode` plus the provider error / structured `badRequestKind` (`session-events.d.ts:2821-2887`). A one-shot fallback scoped to the sub-agent is sufficient.
---
## 6. Appendix
### 6.1 Minimal repro
1. Configure a sub-agent with `CustomAgentConfig.model` = a model that is in the catalog but not onboarded / no-capacity for the test user (``; parent ``).
2. Dispatch it.
3. Observe `model.call_failure (400)` with no parent-model retry; the sub-agent's call fails.
### 6.2 Key references
- `@github/copilot-sdk/dist/types.d.ts:1160-1165` — `CustomAgentConfig.model` fallback contract.
- `@github/copilot-sdk/dist/generated/session-events.d.ts:2821-2887` — `ModelCallFailureEvent` / `ModelCallFailureData` (`ephemeral: true`, `initiator="sub-agent"`, `statusCode`, `model`, `errorMessage`).
- Observed call-time `400`: `Unknown model "". Please ensure the model is onboarded.`
### 6.3 Our runtime-source reading (hypothesis — please verify)
> **Disclaimer:** this is our interpretation of `copilot-agent-runtime`, offered only as a starting pointer. We may be reading the wrong path. Please verify. The ask in §5 stands on the observed behavior (§3 / §4) regardless.
**On why no fallback fires (§3).** Sub-agent model selection appears to run through one of two functions, and both seem to **fail open (no fallback) when the host supplies no populated `available_models` list**:
- `custom_model.rs::resolve_custom_agent_model` — empty `available_models` → first candidate returned verbatim ("CAPI will validate"), no check, no fallback (`:236-260`); non-empty + unresolved → falls back to session model (`:317-328`, the only fallback path we see).
- `agent_tools.rs::validate_and_resolve_subagent_model` — empty → model verbatim (`:171-173`); non-empty + unresolvable → `Error`, no fallback (`:182-190`).
- `available_models` is populated from the CAPI session bootstrap (`model/session.rs:19,96`; `model/capi_client.rs:1587-1628`); an embedded host appears not to feed a catalog, which would leave the list empty on our client.
- Net (our reading): with an empty list the configured model goes straight to CAPI with no check and no fallback → `400`. A populated list may still not fix it — membership is not a capacity check, so an in-list but no-capacity model would still `400` at call-time. This is why we ask for a **call-time** fallback (§5).
**On the foreground path (§4.3).** The source suggests the normal foreground path degrades gracefully: `SessionAgentExecutor.run()` catches a sub-agent turn error and emits `subagent.failed`, returning the failure to the parent as a tool result so the turn continues (`sessionExecutor.ts:282-286`; `session.ts:12364-12382`). We have not verified this end-to-end.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Start with custom_model.rs::resolve_custom_agent_model and agent_tools.rs::validate_and_resolve_subagent_model, then trace the call-time error path through sessionExecutor.ts and session.ts. Reproduce the unavailable-model 400 described in the issue and inspect the ModelCallFailure data in session-events.d.ts. Done means a sub-agent retries once with the parent model after an unavailable-model failure and the turn continues.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- rust, typescript
- Domaine
- ai-infra-agents, backend-api-design
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 52/100