Performance Testing Continues

Tasks ahead:

• update infrastructure/automation in CI

  • determine which VMs/infrastructure to use
  • get automated tests running on appropriate VMs
  • get macOS tests online

• additional tests

  • convert existing functional tests from vcls-test-vscode to use this format and include in CI
  • add some additional in-depth tests for pytorch/bullet3

• Perfecto changes

  • eliminate the 'perf' tests category and convert the existing 'perf' tests to 'functional' tests
  • reduce the noise in the PIT data a bit (trim counter names a bit, especially after dropping perf/func differences)
  • automate typescript instrumentation a bit more (automatically instrument all providers at register* ) - then remove the code from the extension
  • add some simpler higher-level vscode automation functions (harvest from pytorch tests)

Begin by separating the remaining work into CI infrastructure, additional tests, and higher-level VS Code automation. Review the existing vcls-test-vscode functional tests and the pytorch/bullet3 tests, then identify the appropriate VMs and macOS setup. Done means the selected tests run in CI, the additional coverage exists, and the simpler automation functions are available.