`run_factory` should not accept an unconstrained factory name
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
- Copilot Chat Extension Version: Copilot CLI 1.0.78
- VS Code Version: Code - OSS 1.133.0 (development build)
- OS Version: macOS
- Feature (e.g. agent/edit/ask mode): Agents Window / `run_factory` tool
- Selected model (e.g. GPT 4.1, Claude 3.7 Sonnet): GPT-5.6 Sol
- Logs: The raw session event stream recorded 13 model-authored `run_factory` calls. Every call failed immediately with `RpcResponseError: No factory registered with name ""`.
Steps to Reproduce:
1. Expose `run_factory` to an Agents Window session when no matching factories are registered.
2. Ask the agent to perform a routine single-session task, such as a small rendering refinement or analysis of a local screen recording.
3. Observe that the tool schema accepts any string for `name`, allowing the model to invent plausible names and retry them serially.
In the observed session, the model tried eight implementation-oriented names in roughly 49 seconds:
- `apply_finding`
- `code_implementation_factory`
- `single_task`
- `implementation`
- `code-task`
- `fix-factory`
- `repo_task`
- `review_then_implement`
Later it tried five more invented names for video analysis:
- `video_analysis`
- `analyze_video`
- `single_video_analysis`
- `video-analysis`
- `media_analysis`
These were not host retries: each was a distinct assistant-authored tool call after the preceding runtime rejection. No factory ran.
**Expected**
The tool contract should make invalid factory names impossible or strongly bounded. Prefer dynamically constraining `name` to registered factory names. If no factories are registered, do not expose `run_factory`. If dynamic enum generation is not possible, require registry discovery before invocation and return available names in validation errors. The schema should also make `name` required unless `resumeFromRunId` is supplied.
**Actual**
`name` is an optional free-form string, so the model can confuse tool availability with factory availability and repeatedly guess names. The only feedback arrives after each costly tool round trip.
This happened despite session guidance saying orchestration is for non-trivial multi-workstream tasks and repeated delegation failures should fall back to direct work, so prompt guidance alone is not a sufficient guardrail.
Contributor guide
Research direction
Start by locating the Agents Window implementation and the `run_factory` tool schema, then reproduce the issue with no matching factories registered. Trace how registered factory names reach validation and confirm the final behavior prevents invalid calls, handles an empty registry, and requires `name` unless `resumeFromRunId` is supplied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100