Cannot debug E2E tests in VSCode - Dev Server does not seem to run

Start with the VSCode launch.json configuration and the E2E path invoked through node_modules/@stencil/core/bin/stencil; inspect the testing flow referenced in node_modules/@stencil/core/testing/index.js. Reproduce on Windows and compare it with the successful terminal run. Done means debugging the named E2E file starts the dev server on port 3333 and the tests pass.

Stencil version:

 @stencil/core@2.3.0

I'm submitting a:

[x] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://stencil-worldwide.herokuapp.com/ or https://forum.ionicframework.com/

Current behavior:

When debugging an E2E test in VSCode on Windows the tests fail, with request failed errors in the logs. This appears to be due to the dev server not running.

Expected behavior:

E2e tests run successfully whilst debugging through VSCode, to allow adding breakpoints to the tests.

Steps to reproduce:

1. Create new component project through npm init stencil on Windows
2. Open project folder in VSCode
3. Add /.vscode/launch.json (included below)
4. Open my-component.e2e.ts, select the E2E testing profile and hit the play button

Related code:

// launch.json
{
    "configurations": [
      {
        "type": "node",
        "request": "launch",
        "name": "E2E Test Current File",
        "cwd": "${workspaceFolder}",
        "program": "${workspaceFolder}/node_modules/@stencil/core/bin/stencil",
        "args": ["test", "--e2e", "${fileBasename}"],
        "console": "integratedTerminal",
        "internalConsoleOptions": "neverOpen",
        "disableOptimisticBPs": true
      },
      {
        "type": "node",
        "request": "launch",
        "name": "Spec Test Current File",
        "cwd": "${workspaceFolder}",
        "program": "${workspaceFolder}/node_modules/@stencil/core/bin/stencil",
        "args": ["test", "--spec", "${fileBasename}"],
        "console": "integratedTerminal",
        "internalConsoleOptions": "neverOpen",
        "disableOptimisticBPs": true
      }
    ]
  }

(This is a different launch.json from the one recommended in the Stencil documentation, as that one does not work on Windows, see https://github.com/ionic-team/stencil-site/issues/486)

Other information:

Running the Windows Resource Monitor during the e2e test run shows nothing runs on port 3333, where the dev server should be.

The console output from VSCode is:

PS C:\Work\StencilTest\debug-test>  cd 'C:\Work\StencilTest\debug-test'; ${env:NODE_OPTIONS}='--require "c:/Users/me/AppData/Local/Programs/Microsoft VS Code/resources/app/extensions/ms-vscode.js-debug/src/bootloader.bundle.js" --inspect-publish-uid=http'; ${env:VSCODE_INSPECTOR_OPTIONS}='{"inspectorIpc":"\\\\.\\pipe\\node-cdp.23488-6.sock","deferredMode":false,"waitForDebugger":"","execPath":"C:\\Program Files\\nodejs\\node.exe","onlyEntrypoint":false,"autoAttachMode":"always","fileCallback":"C:\\Users\\me\\AppData\\Local\\Temp\\node-debug-callback-75c22f09667419f3"}'; & 'C:\Program Files\nodejs\node.exe' '.\node_modules\@stencil\core\bin\stencil' 'test' '--e2e' 'my-component.e2e.ts'
Debugger attached.
[47:19.1]  @stencil/core
[47:19.4]  v2.3.0
[47:19.5]  testing e2e files
[47:21.5]  build, debug-test, dev mode, started ...
Debugger attached.

Debugger attached.

Debugger attached.

Debugger attached.

Debugger attached.

Debugger attached.

Debugger attached.

[47:21.7]  transpile started ...

[ ERROR ]  stderr: Waiting for the debugger to disconnect... undefined

[47:23.4]  transpile finished in 1.75 s
[47:23.4]  copy started ...
[47:23.4]  generate custom elements bundle started ...
[47:23.4]  generate lazy started ...
[47:23.7]  copy finished (0 files) in 271 ms
[47:24.2]  generate custom elements bundle finished in 771 ms
[47:24.2]  generate lazy finished in 767 ms
[47:24.2]  build finished in 2.73 s

[47:24.3]  jest args: my-component.e2e.ts --e2e --max-workers=8
  console.error
    requestfailed http://localhost:3333/build/debug-test.esm.js

      at node_modules/@stencil/core/testing/index.js:3698:61
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
          at Array.map (<anonymous>)
      at Object.emit (node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
      at Page.emit (node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
      at node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:145:106
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62

  console.error
    Failed to load resource: net::ERR_CONNECTION_REFUSED 
    Location: http://localhost:3333/build/debug-test.esm.js

      at t (node_modules/@stencil/core/testing/index.js:3685:67)
      at node_modules/@stencil/core/testing/index.js:3686:5
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
          at Array.map (<anonymous>)
      at Object.emit (node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
      at Page.emit (node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
      at Page._onLogEntryAdded (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:275:18)

 FAIL  src/components/my-component/my-component.e2e.ts (5.888 s)
  my-component
    × renders (2378 ms)
    × renders changes to the name data (2288 ms)

  ● my-component › renders

    expected to have css class "hydrated"

       7 |     await page.setContent('<my-component></my-component>');
       8 |     const element = await page.find('my-component');
    >  9 |     expect(element).toHaveClass('hydrated');
         |                     ^
      10 |   });
      11 | 
      12 |   it('renders changes to the name data', async () => {

      at Object.<anonymous> (src/components/my-component/my-component.e2e.ts:9:21)

  ● my-component › renders changes to the name data

    Evaluation failed: Error: shadow root does not exist for element: my-component

      at __puppeteer_evaluation_script__:2:30
      at ExecutionContext._evaluateInternal (node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:217:19)
      at l (node_modules/@stencil/core/testing/index.js:816:14)
      at find (node_modules/@stencil/core/testing/index.js:812:33)
      at Object.<anonymous> (src/components/my-component/my-component.e2e.ts:17:21)

Test Suites: 1 failed, 1 total
Tests:       2 failed, 2 total
Snapshots:   0 total
Time:        6.016 s
Ran all test suites matching /my-component.e2e.ts/i.

  console.error
    requestfailed http://localhost:3333/build/debug-test.esm.js

      at node_modules/@stencil/core/testing/index.js:3698:61
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
          at Array.map (<anonymous>)
      at Object.emit (node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
      at Page.emit (node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
      at node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:145:106
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62

  console.error
    Failed to load resource: net::ERR_CONNECTION_REFUSED 
    Location: http://localhost:3333/build/debug-test.esm.js

      at t (node_modules/@stencil/core/testing/index.js:3685:67)
      at node_modules/@stencil/core/testing/index.js:3686:5
      at node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
          at Array.map (<anonymous>)
      at Object.emit (node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
      at Page.emit (node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
      at Page._onLogEntryAdded (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:275:18)

Waiting for the debugger to disconnect...

(It is worth noting that I can successfully debug spec tests through VSCode, so it looks to be an E2E specific problem (which makes sense if it is the dev server that is failing to start))

When the tests are run via the terminal (rather than trying to debug through VSCode) the Windows resource monitor shows the dev server running on port 3333 and the tests all complete successfully (console output with calls to dev server below):

PS C:\Work\StencilTest\debug-test> npm run test

> debug-test@0.0.1 test C:\Work\StencilTest\debug-test
> stencil test --spec --e2e

[06:40.5]  @stencil/core
[06:40.7]  v2.3.0
[06:40.9]  testing e2e and spec files
[06:42.4]  build, debug-test, dev mode, started ...
[06:42.6]  transpile started ...
[06:44.2]  transpile finished in 1.58 s
[06:44.2]  copy started ...
[06:44.2]  generate custom elements bundle started ...
[06:44.2]  generate lazy started ...
[06:44.4]  copy finished (0 files) in 237 ms
[06:44.6]  generate custom elements bundle finished in 463 ms
[06:44.9]  generate lazy finished in 693 ms
[06:44.9]  build finished in 2.45 s

[06:44.9]  jest args: --e2e --spec --max-workers=8
 PASS  src/utils/utils.spec.ts
 PASS  src/components/my-component/my-component.spec.ts
[06:48.1]  GET /build/debug-test.esm.js
[06:48.1]  GET /build/index-e7d8293c.js
[06:48.1]  GET /build/app-globals-0f993ce5.js
[06:48.1]  GET /build/my-component.entry.js
[06:48.4]  GET /build/debug-test.esm.js
[06:48.4]  GET /build/app-globals-0f993ce5.js
[06:48.4]  GET /build/index-e7d8293c.js
[06:48.4]  GET /build/my-component.entry.js
 PASS  src/components/my-component/my-component.e2e.ts

Test Suites: 3 passed, 3 total
Tests:       8 passed, 8 total
Snapshots:   0 total
Time:        3.152 s, estimated 6 s
Ran all test suites.