microsoft / microsoft/vscode

`run_factory` should not accept an unconstrained factory name

Open
#329,551 1 comment 0 reactions 0 assignees View on GitHub
bug chat-agent
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

- Copilot Chat Extension Version: Copilot CLI 1.0.78
- VS Code Version: Code - OSS 1.133.0 (development build)
- OS Version: macOS
- Feature (e.g. agent/edit/ask mode): Agents Window / `run_factory` tool
- Selected model (e.g. GPT 4.1, Claude 3.7 Sonnet): GPT-5.6 Sol
- Logs: The raw session event stream recorded 13 model-authored `run_factory` calls. Every call failed immediately with `RpcResponseError: No factory registered with name ""`.

Steps to Reproduce:

1. Expose `run_factory` to an Agents Window session when no matching factories are registered.
2. Ask the agent to perform a routine single-session task, such as a small rendering refinement or analysis of a local screen recording.
3. Observe that the tool schema accepts any string for `name`, allowing the model to invent plausible names and retry them serially.

In the observed session, the model tried eight implementation-oriented names in roughly 49 seconds:

- `apply_finding`
- `code_implementation_factory`
- `single_task`
- `implementation`
- `code-task`
- `fix-factory`
- `repo_task`
- `review_then_implement`

Later it tried five more invented names for video analysis:

- `video_analysis`
- `analyze_video`
- `single_video_analysis`
- `video-analysis`
- `media_analysis`

These were not host retries: each was a distinct assistant-authored tool call after the preceding runtime rejection. No factory ran.

**Expected**

The tool contract should make invalid factory names impossible or strongly bounded. Prefer dynamically constraining `name` to registered factory names. If no factories are registered, do not expose `run_factory`. If dynamic enum generation is not possible, require registry discovery before invocation and return available names in validation errors. The schema should also make `name` required unless `resumeFromRunId` is supplied.

**Actual**

`name` is an optional free-form string, so the model can confuse tool availability with factory availability and repeatedly guess names. The only feedback arrives after each costly tool round trip.

This happened despite session guidance saying orchestration is for non-trivial multi-workstream tasks and repeated delegation failures should fall back to direct work, so prompt guidance alone is not a sufficient guardrail.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Agents Window implementation and the `run_factory` tool schema, then reproduce the issue with no matching factories registered. Trace how registered factory names reach validation and confirm the final behavior prevents invalid calls, handles an empty registry, and requires `name` unless `resumeFromRunId` is supplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.