wavefnd / wavefnd/Wave

Preserve actual exit codes and failure reasons in Wave case JSON reports

Open
#576 0 comments 0 reactions 0 assignees View on GitHub
bug ci good first issue needs testing
Dominant language
Rust
Stars
53
Forks
16
Avg merge
4h 22m
Merged PRs (30d)
46

Description

## Evidence

[Windows ARM64 cases in PR #574](https://github.com/wavefnd/Wave/actions/runs/34583004765/job/103210726774) report **135 failures**, each with console exit `3221225477` (`0xC0000005`). The JSON builder cannot retain that evidence: nonzero-result branches return `(0, None)`, and the report serializes only name/status when detail is absent.

Sources: [nonzero classification](https://github.com/wavefnd/Wave/blob/ed149e7553c06efc30b5167920cc301563d6afb3/tools/run_tests.py#L380), [JSON serialization](https://github.com/wavefnd/Wave/blob/ed149e7553c06efc30b5167920cc301563d6afb3/tools/run_tests.py#L530).

I reproduced this through the actual `main()` function with one injected case command that prints `compiler failed` to stderr and exits 7. The command correctly fails the suite, but the complete JSON test row is:

```json
{"name": "failed", "status": "fail"}
```

The test command used a real Python subprocess; only discovery/compiler selection was injected. This source-audit reproduction is independent of the native compiler crash.

## Acceptance

- [ ] Failed rows retain actual and expected exit status and an actionable reason, including native crash codes without truncation.
- [ ] Artifact-contract failures retain their reason rather than only `status: fail`.
- [ ] Timeout and skip reasons continue to work; existing summary/name/status fields remain compatible.
- [ ] Focused fake-command tests validate serialized failure rows. Keep output excerpts bounded if included.

This is about per-test failure evidence. #462 separately covers target/suite identity, and #369 covers expected-output fixtures. Do not duplicate those tasks or claim this fixes #493.

Contributor guide

Open the contributing guide

Research direction

Start in tools/run_tests.py at the nonzero-result classification around line 380 and JSON serialization around line 530, then trace the behavior through main(). Add focused fake-command coverage for serialized failure rows, including exit status and failure reasons, while preserving existing timeout, skip, summary, name, and status fields. Run the relevant test-runner checks and confirm native crash codes are not truncated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.