Azure-Samples / Azure-Samples/azuresandbox

Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD

Open
#498 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
PowerShell
Stars
154
Forks
75
Avg merge
15h 2m
Merged PRs (30d)
45

Description

## Summary

Prerequisite for #173 (CD) — specifically for precise, per-check test-failure surfacing in GitHub Actions.

`scripts/Invoke-UnitTests.ps1` today only exposes:

- An **exit code** (`0` pass / `1` test failures / `2` harness error).
- A human-readable **log file** + stdout containing per-check `[PASS]`/`[FAIL] ` lines and a
per-module `[SUMMARY] Passed: N Failed: N Total: N` line.
- A final per-**module** summary table; the orchestrator keeps only per-module counts in memory
(`$moduleResults`) — there is **no machine-readable artifact** and no structured per-check result.

This makes it hard for the CD workflow to pinpoint the exact failing assertion: it would have to
regex-parse free-text log lines (brittle).

## Goal

Emit a **stable, machine-readable results file** so CI/CD can render per-check failures robustly
(GitHub Job Summary table, `::error::` annotations, test-reporter actions).

## Acceptance criteria

- [ ] Optional output path parameter(s) (e.g. `-ResultsJsonPath` and/or `-JUnitXmlPath`) that, when
supplied, write structured results without changing default behavior.
- [ ] **JSON** output capturing, at minimum: overall pass/fail + counts, and per-module results with
a per-**check** breakdown (check name, status, and message/detail for failures).
- [ ] **JUnit XML** output (testsuites/testcase, with `` for failed checks) so standard
GitHub test-reporter actions can consume it.
- [ ] Per-check detail is captured at the orchestrator level (not just per-module counts) — the
per-module test scripts surface individual check results to the orchestrator.
- [ ] Existing exit codes, log file, and console output remain unchanged (backward compatible).
- [ ] README / header comments document the new outputs.

## Notes

- Consumed by the #173 CD workflow's `workflow-tests` step to build a `$GITHUB_STEP_SUMMARY` table
and emit annotations for each failed check, plus upload the JSON/JUnit as artifacts.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/Invoke-UnitTests.ps1 and trace how per-module results, $moduleResults, exit codes, logs, and console output are produced; run the existing unit-test invocation to establish its current behavior. Done means optional JSON and JUnit outputs contain overall, module, and per-check results while existing outputs remain unchanged, with README or header documentation covering the new paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, powershell
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.