feat(reporting): distinguish "not applicable" from "could not verify" and make unproven checks fail the run
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8
- Forks
- 1
- Avg merge
- 18h 23m
- Merged PRs (30d)
- 63
Description
VIP's value is the claim that a deployment was validated. The most dangerous failure mode for a validation tool is not a red run — it is a green run that proved nothing. Today those two outcomes are indistinguishable: a check that was legitimately N/A and a check that could not be executed both land in the same skipped bucket, and neither affects the exit code.
Evidence that this is a class, not a one-off
#596 (configured product never authenticates, every test skips, exit 0), #602 (test_workbench_login skips under SSO, so no lane verifies a Workbench login at all), #606 (test_connect_login_ui can never fail — its only assertion is satisfied before credentials are submitted), #550 and #546 (suites that cannot run in CI because [runtimes] is unpopulated and nothing terminates TLS). Each has been triaged individually; nothing prevents the next one.
Structural evidence
src/vip_tests contains 160 pytest.skip() call sites. ReportData (src/vip/reporting.py:90) counts skips as a single undifferentiated total, and the only classification that exists is na_version. Exit status is pytest's raw exitstatus (src/vip/plugin.py:1266), so a run in which every meaningful check skipped exits 0 and renders a report with zero failures. To someone reading that report as an audit artifact, "we did not check" and "we checked and it is fine" look identical.
Proposal
Introduce a third outcome, unproven, sitting between passed and failed.
- Classify at the skip site.
vip.attest.not_applicable(reason)marks a skip as deliberate and non-fatal (product not configured, tier lacks the feature, IDE not installed) and keeps it out of the exit code;vip.attest.unproven(reason)marks one VIP was asked to run and could not. A barepytest.skip()keeps thenot_applicablemeaning, so sites convert deliberately rather than by flag day — defaulting to unproven would make "product not configured" fatal, which is exactly the case #596's note on the reverse risk says must stay non-fatal. - Surface it everywhere the outcome is reported: report counters and badges, JUnit (
unprovenas a distinct message), SARIF (warning, betweennoneanderror). - Make it affect the exit code, behind
--allow-unprovenfor anyone who needs the old behaviour. - Split the report's top line three ways — Verified / Failed / Not verified — with "not verified" broken into "N/A for this version" and "could not run here".
The tradeoff, stated plainly
This makes VIP louder on day one. Existing lanes that quietly skipped will start failing, and the first run against a real deployment will surface gaps nobody has had to look at. That is the intended effect, but it is a real behaviour change and the escape hatch exists for that reason.
Why it is worth it
A tool that reports "31 of your 47 configured capabilities were verified, here are the 16 I could not reach and why" is more credible as an IQ/OQ artifact than one that reports 47/47 green with the interesting half silently skipped. #596 already names the required distinction — "separate 'nothing to do here' from 'we could not do what was asked'" — this issue supplies the mechanism, and closes #596 as its first consumer.
Scope note
Landing in slices: this issue covers the unproven outcome, its plumbing through the report formats, and the exit-code contract. Retriaging the existing 160 skip sites is follow-up work, as is the capability-attestation layer (asserting that a declared capability had a check that actually exercised it) and a mutation meta-test that proves each scenario can fail.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ReportData in src/vip/reporting.py:90 and exit-status handling in src/vip/plugin.py:1266, then trace the 160 pytest.skip() sites under src/vip_tests. Map the new unproven outcome through report counters, badges, JUnit, SARIF, and the exit-code path, including --allow-unproven. Done means unproven is distinct from not applicable, appears in all specified outputs, and affects the run status as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100