microsoft / microsoft/vscode

Re-enable Chat Sandbox smoke tests in Remote runs

Open
#332,986 1 comment 0 reactions 1 assignee Claimed by @dileepyavan View on GitHub
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Summary

Chat Sandbox smoke tests have been temporarily disabled for Remote smoke-test runs in PR #332892.

The suite itself was previously registered for every non-web run, including Remote. However, the `vscode-smoketest-ext-host` helper extension is copied into the smoke-test extensions directory only for non-web, non-Remote runs. That extension contributes `smoketest.openLocalChat`, which the Chat Sandbox suite invokes in its `before` hook to open the built-in Local test harness.

As a result, `smoketest.openLocalChat` is not present in the Command Palette during Remote runs.

## Why this appeared to work before

Before PR #332892, `QuickAccess.runCommand()` typed the requested command ID into the Command Palette but accepted any focused result unless the row said `No matching commands`. When `smoketest.openLocalChat` was absent, the Command Palette returned another fuzzy chat-related result and automation accepted it. This masked the missing helper command.

PR #332892 makes command automation verify the selected item's exact command ID by default. Linux and macOS Remote smoke tests then failed deterministically with:

```
Error: QuickAccess.runCommand(command: smoketest.openLocalChat, match: exactCommandId) failed to find command.
```

Returning this call to fuzzy matching would hide the setup problem and could execute an unrelated command, so the suite is temporarily gated with `!opts.remote` instead.

## How Remote coverage could be re-enabled

Possible approaches:

1. Make `vscode-smoketest-ext-host` available in Remote smoke runs. It declares `extensionKind: ["ui"]`, so the setup should ensure it is installed in the correct local/UI extension location and that `smoketest.openLocalChat` is registered before the suite invokes it.
2. Replace the helper with exact UI-visible commands and interactions that select the Local test harness. The underlying `workbench.action.chat.newLocalChat` command currently has `f1: false`, which is why the helper command exists.
3. If Chat Sandbox is intentionally unsupported in Remote configurations, keep the exclusion and document that limitation explicitly.

## Acceptance criteria

- Re-enable `setupChatSandboxTests(logger)` for Remote smoke runs.
- Ensure the requested command or replacement UI flow exists in Remote and is selected using exact matching.
- Verify the Chat Sandbox suite passes in Linux and macOS Remote smoke jobs without relying on an unrelated fuzzy Command Palette result.

Related: #332892

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.