bowtie-json-schema / bowtie-json-schema/bowtie

Uncaught harness errors should cause non-zero exit statuses

Open
#809 19 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
93
Forks
77
Avg merge
10h 29m
Merged PRs (30d)
40

Description

A `bowtie suite` or `bowtie run` execution treats test failures by still exiting with 0 exit code, as failures are part of Bowtie's "normal behavior" as a compliance checker.

However -- *errors*, particularly uncaught errors which should be being handled in the harness, are not, and should likely trigger nonzero exit status.

Note that as part of fixing this you may also need to check whether any existing harnesses have uncaught errors (and to catch them if so) -- otherwise Bowtie's own report generation will begin to fail.

Contributor guide

Open the contributing guide

Research direction

Start with the `bowtie suite` and `bowtie run` entry points and trace how test failures and uncaught harness errors determine the process exit status. Check existing harnesses for uncaught errors and verify that Bowtie's report generation still succeeds. Done means uncaught harness errors produce a non-zero exit status while normal test failures retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.