Re-enable Chat Sandbox smoke tests in Remote runs
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
## Summary
Chat Sandbox smoke tests have been temporarily disabled for Remote smoke-test runs in PR #332892.
The suite itself was previously registered for every non-web run, including Remote. However, the `vscode-smoketest-ext-host` helper extension is copied into the smoke-test extensions directory only for non-web, non-Remote runs. That extension contributes `smoketest.openLocalChat`, which the Chat Sandbox suite invokes in its `before` hook to open the built-in Local test harness.
As a result, `smoketest.openLocalChat` is not present in the Command Palette during Remote runs.
## Why this appeared to work before
Before PR #332892, `QuickAccess.runCommand()` typed the requested command ID into the Command Palette but accepted any focused result unless the row said `No matching commands`. When `smoketest.openLocalChat` was absent, the Command Palette returned another fuzzy chat-related result and automation accepted it. This masked the missing helper command.
PR #332892 makes command automation verify the selected item's exact command ID by default. Linux and macOS Remote smoke tests then failed deterministically with:
```
Error: QuickAccess.runCommand(command: smoketest.openLocalChat, match: exactCommandId) failed to find command.
```
Returning this call to fuzzy matching would hide the setup problem and could execute an unrelated command, so the suite is temporarily gated with `!opts.remote` instead.
## How Remote coverage could be re-enabled
Possible approaches:
1. Make `vscode-smoketest-ext-host` available in Remote smoke runs. It declares `extensionKind: ["ui"]`, so the setup should ensure it is installed in the correct local/UI extension location and that `smoketest.openLocalChat` is registered before the suite invokes it.
2. Replace the helper with exact UI-visible commands and interactions that select the Local test harness. The underlying `workbench.action.chat.newLocalChat` command currently has `f1: false`, which is why the helper command exists.
3. If Chat Sandbox is intentionally unsupported in Remote configurations, keep the exclusion and document that limitation explicitly.
## Acceptance criteria
- Re-enable `setupChatSandboxTests(logger)` for Remote smoke runs.
- Ensure the requested command or replacement UI flow exists in Remote and is selected using exact matching.
- Verify the Chat Sandbox suite passes in Linux and macOS Remote smoke jobs without relying on an unrelated fuzzy Command Palette result.
Related: #332892
Contributor guide
Assessment
This issue has not been assessed yet.