posit-dev / posit-dev/vip

feat(reporting): distinguish "not applicable" from "could not verify" and make unproven checks fail the run

Open
#616 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
8
Forks
1
Avg merge
18h 23m
Merged PRs (30d)
63

Description

VIP's value is the claim that a deployment was validated. The most dangerous failure mode for a validation tool is not a red run — it is a green run that proved nothing. Today those two outcomes are indistinguishable: a check that was legitimately N/A and a check that could not be executed both land in the same skipped bucket, and neither affects the exit code.

Evidence that this is a class, not a one-off

#596 (configured product never authenticates, every test skips, exit 0), #602 (test_workbench_login skips under SSO, so no lane verifies a Workbench login at all), #606 (test_connect_login_ui can never fail — its only assertion is satisfied before credentials are submitted), #550 and #546 (suites that cannot run in CI because [runtimes] is unpopulated and nothing terminates TLS). Each has been triaged individually; nothing prevents the next one.

Structural evidence

src/vip_tests contains 160 pytest.skip() call sites. ReportData (src/vip/reporting.py:90) counts skips as a single undifferentiated total, and the only classification that exists is na_version. Exit status is pytest's raw exitstatus (src/vip/plugin.py:1266), so a run in which every meaningful check skipped exits 0 and renders a report with zero failures. To someone reading that report as an audit artifact, "we did not check" and "we checked and it is fine" look identical.

Proposal

Introduce a third outcome, unproven, sitting between passed and failed.

  1. Classify at the skip site. vip.attest.not_applicable(reason) marks a skip as deliberate and non-fatal (product not configured, tier lacks the feature, IDE not installed) and keeps it out of the exit code; vip.attest.unproven(reason) marks one VIP was asked to run and could not. A bare pytest.skip() keeps the not_applicable meaning, so sites convert deliberately rather than by flag day — defaulting to unproven would make "product not configured" fatal, which is exactly the case #596's note on the reverse risk says must stay non-fatal.
  2. Surface it everywhere the outcome is reported: report counters and badges, JUnit (unproven as a distinct message), SARIF (warning, between none and error).
  3. Make it affect the exit code, behind --allow-unproven for anyone who needs the old behaviour.
  4. Split the report's top line three ways — Verified / Failed / Not verified — with "not verified" broken into "N/A for this version" and "could not run here".

The tradeoff, stated plainly

This makes VIP louder on day one. Existing lanes that quietly skipped will start failing, and the first run against a real deployment will surface gaps nobody has had to look at. That is the intended effect, but it is a real behaviour change and the escape hatch exists for that reason.

Why it is worth it

A tool that reports "31 of your 47 configured capabilities were verified, here are the 16 I could not reach and why" is more credible as an IQ/OQ artifact than one that reports 47/47 green with the interesting half silently skipped. #596 already names the required distinction — "separate 'nothing to do here' from 'we could not do what was asked'" — this issue supplies the mechanism, and closes #596 as its first consumer.

Scope note

Landing in slices: this issue covers the unproven outcome, its plumbing through the report formats, and the exit-code contract. Retriaging the existing 160 skip sites is follow-up work, as is the capability-attestation layer (asserting that a declared capability had a check that actually exercised it) and a mutation meta-test that proves each scenario can fail.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ReportData in src/vip/reporting.py:90 and exit-status handling in src/vip/plugin.py:1266, then trace the 160 pytest.skip() sites under src/vip_tests. Map the new unproven outcome through report counters, badges, JUnit, SARIF, and the exit-code path, including --allow-unproven. Done means unproven is distinct from not applicable, appears in all specified outputs, and affects the run status as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.