GitHub Actions: PR-Optimized Test Evidence Comment Renderer

Context

We already have a working Test Evidence pipeline with the following properties:

• JSON is the single source of truth

  • test-specification.preview.json
  • test-specification.pr.json (diff JSON)

• A local Diff MD renderer exists and is useful for developer-side inspection

• Diff logic is stable and based on structured JSON comparison (not MD diff)

The next step is to integrate this into GitHub Actions so that Test Evidence is automatically generated and posted to Pull Requests in a reviewer-friendly way.

---

Design Principles (Non-Negotiable)

• Diff calculation must remain JSON-based

• Rendering must be separated from diff logic

• Multiple MD outputs are allowed as projections, not as independent sources

• No duplicated diff logic

• GitHub PR comments should be:

  • Quiet (no noise)
  • Mechanical (no human judgment text)
  • Clearly indicate whether evidence exists or not

---

Goal

Introduce a GitHub-optimized renderer for Test Evidence and wire it into GitHub Actions.

This renderer is specifically for PR comments and may differ from the existing local Diff MD format.

---

Scope

1. Core remains unchanged

Do not modify:

• Diff JSON schema
• Diff calculation logic

The following function is considered the single source of truth:

buildDiffJson(basePreviewJson, headPreviewJson, options) -> diffJson

---

2. Introduce a new renderer for GitHub PR comments

Add a new renderer function, for example:

renderDiffForGitHubPR(diffJson, options)
  -> { mode: "full" | "short" | "skip", body?: string }

This renderer is GitHub-specific and separate from the local MD renderer.

---

3. Rendering behavior rules (PR comment)

Case A: Diff exists (normal case)

Conditions:

• Any of catalogs/tests has + / - / ~

Behavior:

• Post a full PR comment

• Content is derived from diff JSON

• Format may include:

  • Summary header
  • Optional <details> folding (to be refined later)

Example summary line (conceptual):

Test Evidence: +1 catalog, -1 catalog, ~2 catalogs; +3 tests, -2 tests, ~1 test.

---

Case B: No diff

Conditions:

• catalogs: +0 / -0 / ~0
• tests: +0 / -0 / ~0

Behavior:

• Do not post the full report
• Post a short status comment only

Example:

Test Evidence: no changes detected.

---

Case C: No supported catalogs / evidence not applicable

Conditions (examples):

• head totals: catalogs=0 tests=0
• Or evidence collection reports “no supported test catalogs”

Behavior:

• Do not post the full report
• Post a short explanatory comment

Example:

Test Evidence: no supported test catalogs found in this PR.

---

Case D: Generation failure

Conditions:

• Diff generation failed
• Base preview could not be generated

Behavior:

• Post a failure status comment only
• No partial or misleading output

Example:

Test Evidence: failed to generate.

---

4. Comment update policy

• The Action must update a single comment, not create new comments repeatedly
• Use a stable identifier (bot user + hidden marker) to find and update the comment
• Prevent comment spam

---

5. GitHub Actions integration

This Issue does not require full YAML authoring, but assumes:

• Triggered on Pull Requests

• Renderer output decides:

  • full comment
  • short comment
  • or skip (no comment at all)

The Action should:

1. Generate preview JSON (base/head)
2. Build diff JSON
3. Invoke GitHub PR renderer
4. Post/update comment according to renderer result

---

Non-Goals (Explicitly Out of Scope)

• Migrating existing tests to catalog format
• AST-based migration tooling
• Public package extraction
• HTML or web UI rendering
• Snapshot support

These may be future work.

---

Why this approach

• Keeps JSON as the only authoritative data
• Avoids double maintenance of diff logic
• Allows local MD and GitHub PR views to evolve independently
• Makes CI output predictable and machine-trustable
• Reduces PR noise while preserving confidence

---

Acceptance Criteria

• A new GitHub-specific renderer exists
• Renderer behavior matches the rules above
• No duplicate diff logic introduced
• PR comments are updated, not spammed
• Short “no diff / not applicable” comments are visible and unambiguous

---

This Issue should be sufficient for the next AI or developer to implement the feature end-to-end.

Start by locating the buildDiffJson entry point, the existing local Diff MD renderer, and the test-specification.preview.json and test-specification.pr.json generation flow. Then inspect the GitHub Actions integration and determine how PR comments are currently posted. Done means a separate GitHub renderer handles full, short, and failure statuses while updating one stable comment without changing diff logic.