Reject missing object outputs in manifest compile-only case runs
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 4h 22m
- Merged PRs (30d)
- 46
Description
## Problem and reproduction
A manifest-selected compile-only lane can report PASS when the compiler exits zero without producing any object.
`tools/run_tests.py::command_for_test` requests `--emit=obj` for `manifest_compile_target()`. In `run_and_classify`, however, artifact validation runs only when `compile_target is None` (lines 405-412), so the cross-target branch checks the process exit code alone.
I executed the actual checked-in `run_and_classify` function in an isolated namespace with:
- a selected `aarch64-unknown-freebsd` compile target;
- ordinary successful metadata;
- `[sys.executable, "-c", "pass"]` as the fake compiler command.
The fake command creates no output. The function prints `PASS` and returns `(1, None)`.
## Scope and acceptance
- [ ] Require the expected object file to exist before a manifest compile-only case is counted as passed.
- [ ] Derive the output location from the same source/name/output-directory logic used to construct the command, including `testN/main.wave` cases.
- [ ] Add fake-compiler tests for exit-zero/missing-file, nonzero exit and a successful output.
- [ ] Use existing artifact checks where applicable and preserve host/target selection.
- [ ] Keep adding new object-format validators out of this small fix; #463 already tracks WebAssembly format validation.
No LLVM or native cross-platform machine is needed for the focused Python tests. This is a good first issue in test result classification, separate from #506's VM discovery and #462's report metadata.
Audited on canonical master `0c67f4cc0c3946cbf708c11ef79db927ec8f054e` (same source tree as #502 head). Executable reproductions used Fedora Linux amd64, Wave `0.2.1-pre-beta-dev`, LLVM 21.1.8.
Contributor guide
Research direction
Start in tools/run_tests.py with command_for_test and run_and_classify, then trace manifest_compile_target() and the existing artifact checks. Add focused fake-compiler tests covering exit-zero with no object, nonzero exit, and successful output, including testN/main.wave paths. Done means missing expected objects cannot pass while host/target selection remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100