Surface `run_tests.py` diagnostics in `alloy.py` build logs when tests fail
- Dominant language
- C++
- Stars
- 3k
- Forks
- 352
- PR merge metrics
- No merged PRs in 30d
Description
When a test job launched from `alloy.py` fails, the build log only contains a high-level summary (the `STABILITY REPORT`) followed by a diff of `fail_db.txt` listing the newly failing tests. For example:
```
_________________________STABILITY REPORT_________________________
Running native - AVX2
Using test compiler: Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.31.0dev (build commit 2857de55446851ee @ 20260622, LLVM 21.1.8)
Using C/C++ compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35228 for x64
x86-64 generic-i32x4 unspec Windows LLVM 21.1 cl O2 obj. Fails: 12, New fails: 12. 00h09m44sec.
NO NEW RUNFAILS
NEW COMPFAILS: 12
NO NEW PASSES RUNFAILS
NO NEW PASSES COMPFAILS
```
The summary tells you *which* tests failed, but not *why*. The actual diagnostics from `run_tests.py` (compiler errors, mismatched output, stderr, etc.) are not included in the log, so there is no straightforward way to understand the root cause from the failed job alone.
**Problem:**
Finding the real diagnostics is non-obvious and time-consuming. To investigate a failure you need to dig through artifacts manually, even though `run_tests.py` already produced the detailed output during the run.
**Suggested improvement:**
Make the per-test diagnostics from `run_tests.py` easily discoverable from the failed `alloy.py` job — for example, by including the failing tests' compile/run output in the log (or as a saved artifact) alongside the stability report.
Contributor guide
Research direction
Start at alloy.py's test-job log handling and trace the diagnostics emitted by run_tests.py; inspect how failed-test artifacts are currently exposed. Reproduce a failing job, then make the per-test compiler/run output discoverable in the build log or as a saved artifact alongside the STABILITY REPORT, and verify the failed job includes the diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, testing, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100