microsoft / microsoft/playwright
[Feature] Define different exit codes to make CI/CD scripting easier
- Dominant language
- TypeScript
- Stars
- 96.3k
- Forks
- 6.5k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 180
Description
It looks like `process.exit(1);` is used universal throughout the project when something does not go as expected. `process.exit(1);` is also used when everything does execute properly, but not all test cases passed. Finally, it seems there are a couple cases were [exit code 0 is associated with an error](https://github.com/microsoft/playwright/blob/729e41b3aa7aa67244c6e4eb8195a328034913d7/packages/playwright-core/src/cli/cli.ts#L414).
It would be very handy for CI/CD scripting to easily determine Playwright's exit reason. For very basic support, it could be:
* `0`: Everything executed properly with all tests passing
* `1`: Everything executed properly, but with failing tests
* `2`: Something unexpected happened during execution
For my needs, these three different exit code would be good enough. I would like to run tests in the pipeline - and I have some tests that will periodically fail due to external reasons. As long as all tests are running, I would like the pipeline to be successful - or perhaps marked as unstable. On the other hand, if something new was introduced that is causing the tests to break - then that is a problem to halt the pipeline over.
Depending on how granular you want to go, you could add even more exit codes. This is what [pytest](https://docs.pytest.org/en/7.1.x/reference/exit-codes.html) does for their exit codes:
> * `0`: All tests were collected and passed successfully
> * `1`: Tests were collected and run but some of the tests failed
> * `2`: Test execution was interrupted by the user
> * `3`: Internal error happened while executing tests
> * `4`: pytest command line usage error
> * `5`: No tests were collected
I'm not opposed to trying to add support for more consistent exit codes myself, but I want to make sure everyone is aligned on expected behavior before any time is spent on it.
Thanks!
Contributor guide
Research direction
Start with the exit-code handling in packages/playwright-core/src/cli/cli.ts, especially the linked location, and search the project for the other process.exit(1) and process.exit(0) cases mentioned in the issue. The work is not ready to implement until the expected meanings for each exit code are agreed; done would mean consistent behavior for the agreed execution outcomes and coverage for those cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100