devcontainers / devcontainers/cli

Ability to run a specific global scenario test

Open
#227 2 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
TypeScript
Stars
3k
Forks
457
Avg merge
13h 17m
Merged PRs (30d)
6

Description

Hello,

Following on from #163 I converted all my features which use `installsAfter` to use the [global scenarios](https://github.com/virtualstaticvoid/devcontainers-cli-issue-163/pull/1/commits/5d8b80d1669e458cd7a1d40776548781e945a0c0) test functionality and [removed](https://github.com/virtualstaticvoid/devcontainers-cli-issue-163/pull/1/commits/1719c646dc94b45f82ebdd6552cc563333917ed1) them from specific feature tests since they always fail for the default test.

Whilst in my example for #163 I only had one feature, my work project has over 20 features.

The issue now is that the GitHub Actions build times out (>6 hours) running the [global scenarios](https://github.com/virtualstaticvoid/devcontainers-cli-issue-163/blob/main/.github/workflows/test.yaml#L27-L28) for 20+ features.

This issue could be solved in one of the following ways:

1. Allow running a specific global scenario test, providing it's name on the command line, such as with a command line switch `--global-scenario=`.
This would enable use of a build matrix for global scenarios too, such as the [one used in `feature-template`](https://github.com/devcontainers/feature-template/blob/main/.github/workflows/test.yaml#L13-L21) to run specific feature tests individually. This would also aid debugging failed scenarios and allow them to be run in parallel.

2. Use a `scenarios.json` within the features's test directory. If the file is present, then the "default" feature test should _not_ run, since it won't pass anyway, and the included scenarios be run instead. Global scenarios wouldn't be needed in this case.

I would think option 2 would be the preferred approach, as it shouldn't break any existing implementations and as I understand global scenarios are intended for testing whether a combination of features work together instead of for features which simply use `installsAfter`. Having a `scenarios.json` file within a features test directory seems more obvious and natural place.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.