ispc / ispc/ispc

Surface `run_tests.py` diagnostics in `alloy.py` build logs when tests fail

Open
#3,868 0 comments 2 reactions 0 assignees View on GitHub
CI / GitHub Actions
Dominant language
C++
Stars
3k
Forks
352
PR merge metrics
No merged PRs in 30d

Description

When a test job launched from `alloy.py` fails, the build log only contains a high-level summary (the `STABILITY REPORT`) followed by a diff of `fail_db.txt` listing the newly failing tests. For example:

```
_________________________STABILITY REPORT_________________________
Running native - AVX2

Using test compiler: Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.31.0dev (build commit 2857de55446851ee @ 20260622, LLVM 21.1.8)
Using C/C++ compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35228 for x64

x86-64 generic-i32x4 unspec Windows LLVM 21.1 cl O2 obj. Fails: 12, New fails: 12. 00h09m44sec.
NO NEW RUNFAILS
NEW COMPFAILS: 12
NO NEW PASSES RUNFAILS
NO NEW PASSES COMPFAILS
```

The summary tells you *which* tests failed, but not *why*. The actual diagnostics from `run_tests.py` (compiler errors, mismatched output, stderr, etc.) are not included in the log, so there is no straightforward way to understand the root cause from the failed job alone.

**Problem:**

Finding the real diagnostics is non-obvious and time-consuming. To investigate a failure you need to dig through artifacts manually, even though `run_tests.py` already produced the detailed output during the run.

**Suggested improvement:**

Make the per-test diagnostics from `run_tests.py` easily discoverable from the failed `alloy.py` job — for example, by including the failing tests' compile/run output in the log (or as a saved artifact) alongside the stability report.

Contributor guide

Open the contributing guide

Research direction

Start at alloy.py's test-job log handling and trace the diagnostics emitted by run_tests.py; inspect how failed-test artifacts are currently exposed. Reproduce a failing job, then make the per-test compiler/run output discoverable in the build log or as a saved artifact alongside the STABILITY REPORT, and verify the failed job includes the diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, testing, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.