Parallel explore subagent fan-out dies to per-model 429s: explore's default model is the only rate-limited one, no backoff, no auto model switch despite eligibleForAutoSwitch
- Dominant language
- Shell
- Stars
- 11.2k
- Forks
- 1.9k
- Avg merge
- 14h 16m
- Merged PRs (30d)
- 6
Description
### Describe the bug
Launching many subagents in parallel via the task tool concentrates all their model calls on one model bucket — `explore` agents all default to the same lightweight model (currently claude-haiku-4.5). That model appears to have a much tighter per-model burst limit than any other model, so a 16-agent explore fan-out hits HTTP 429 within ~20 seconds. Every subagent then fails with repeated `integration_rate_limited` errors and completes with empty output, while the parent session (on a different model) continues unaffected.
Three compounding problems:
1. **The built-in `explore` agent defaults to the only model that rate-limits under fan-out.** A survey of 1,089 local CLI sessions (~2.7 GB of events.jsonl) found exactly 5 sessions that ever logged `errorType: rate_limit` — every significant incident was a 16x claude-haiku-4.5 explore fan-out. Other models never triggered it despite far heavier use: gpt-5.6-sol (19k assistant messages, 201 sessions), claude-fable-5 (14k messages, including a clean 16-way fan-out re-run of the exact workload that failed on haiku), claude-sonnet-5 (9k), claude-opus-5 (7k), gpt-5.6-terra (5k) — zero incidents.
2. **No backoff.** The agentic loop retried into the same window at ~2 failing requests/second: 182 `integration_rate_limited` errors logged in 87 seconds in one incident.
3. **No model fallback.** The 429 response carries `"eligibleForAutoSwitch": true`, but subagents never switch — they just die and return empty results, silently wasting the whole fan-out.
Sample error from events.jsonl:
```json
{"errorType":"rate_limit",
"message":"You've hit the rate limit for this model. Please switch models or wait for your limit to reset in under a minute. Learn More (https://docs.github.com/copilot/concepts/rate-limits). (Request ID: C33A:2A98BA:2871A5:13FF8A5:6A7702DD)",
"statusCode":429,
"errorCode":"integration_rate_limited",
"eligibleForAutoSwitch":true}
```
Observed concurrency data for the explore default model (same account, 4 incidents across 3 days):
- 16 parallel explore agents -> first 429 in ~20 s, all 16 stall and return empty
- 8 parallel -> clean
- ~15 launched sequentially over 10 minutes with overlap -> 2 transient errors, work completed
- 16 parallel on claude-fable-5 (model override) -> zero errors
### Affected version
1.0.79-5 (Windows x64); incidents also observed on earlier 1.0.7x builds.
### Steps to reproduce the behavior
1. In a session, use the task tool to launch ~16 `explore` subagents simultaneously (all default to the lightweight model).
2. Within ~20 seconds, subagent transcripts fill with "Limit reached — Resets in under a minute"; agents go idle after emitting only setup text and return empty results.
3. Re-run the identical fan-out with a `model` override to a larger model — it completes cleanly.
### Expected behavior
Any (ideally all) of:
- Subagents honor `eligibleForAutoSwitch` and fall back to another model instead of dying (related: #2840).
- The agentic loop backs off per the reset window instead of retrying ~2x/s into the same limited minute (related: #2760).
- The CLI throttles subagent fan-out concurrency per model client-side, since it knows how many concurrent loops it is aiming at one bucket (related: #2545).
- The built-in `explore` agent''s default model either gets burst headroom matching the "fast, lightweight, fan out in parallel" positioning, or the CLI spreads large fan-outs across multiple eligible lightweight models.
### Additional context
- The failure is invisible from the parent''s perspective until results come back empty: `subagent.completed` events fire normally with ~200-byte payloads.
- Nudging stalled subagents (`write_agent`) during the limited minute makes it worse — each nudge adds more 429s to the same window.
- Request IDs from multiple incidents available on request.
Contributor guide
Research direction
Start by reproducing the failure with the task tool using a 16-agent explore fan-out, then compare events.jsonl for the default model and a model override. Done should prevent repeated 429 retries and ensure subagents return useful results through backoff, throttling, or model switching; the payload names no implementation file or test.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai-infra-agents, cli, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100