microsoft / microsoft/vscode

Agents window: enable signed-out use by default via experimentation

Open
#329,671 0 comments 5 reactions 1 assignee Claimed by @TylerLeonhardt View on GitHub
agents-window feature-request
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

Follow-up to #318578 and #328990.

## Goal

Roll out the signed-out Agents window behavior through experimentation by enabling `chat.agentHost.allowSignedOutWhenUsable` for treatment users.

#328990 added the platform capability, but the setting remains experimental and off by default. This issue tracks validating the experience and making it the default behavior through the experimentation system rather than requiring users to opt in manually.

## Treatment behavior

When the user is signed out of GitHub and at least one registered session type can run using its own credentials:

- Open the Agents window without showing the blocking GitHub sign-in flow.
- Keep GitHub sign-in available as an optional action for users who want Copilot-backed models or functionality.
- Request GitHub authentication on demand when a selected provider, model, or operation requires it.

When no registered session type is usable without GitHub, retain the existing sign-in/setup experience.

## Rollout plan

- Configure an experiment whose treatment enables `chat.agentHost.allowSignedOutWhenUsable`.
- Keep the current behavior as the control.
- Validate the signed-out entry, provider selection, model selection, account, and authentication-on-demand experiences.
- Monitor sign-in conversion, successful session creation and first-turn completion, auth failures, provider availability, and error rates.
- Confirm the experiment can be rolled back without leaving the Agent Host and workbench in inconsistent states.
- After successful validation, graduate the behavior so users no longer need to set the experimental setting manually.

## Readiness / related work

- #329007 — avoid flashing the blocking sign-in modal while provider availability is still resolving.
- #329667 — allow Copilot Agent Host BYOK sessions to participate while signed out.
- #329656 — Claude native/Copilot provider auth behavior.
- #329668 — Codex non-Copilot provider auth behavior.

## Acceptance criteria

- Treatment users receive the signed-out behavior without manually changing settings.
- Control users retain the current behavior.
- Users with no usable signed-out provider are not left in an empty or unusable Agents window.
- Provider and account changes update the gate reactively.
- Experiment metrics and rollback criteria are documented before broad rollout.
- The validated behavior is graduated from the experimental opt-in.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.