BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(test-writer): platform-specific tests emitted without consulting CI runner OS — Linux-only failure injection silently broken on macOS-pinned CI

Open
#359 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

test-writer can emit a **platform-specific test** (e.g. a Linux-only I/O-failure path) without consulting the project's **actual CI runner OS**. When the runner OS differs (project CI pinned to `macos-latest`), the test is non-deterministic or outright broken on the only platform that gates merges — yet it passes test-authoring and the Red Gate because those run wherever the author happens to be.

## Concrete instance (generalized)

A story needed a real-I/O write-failure test to exercise an error mapping (open/write → `Io(_)`). test-writer reached for the classic Linux trick:

```rust
// write to /dev/full → ENOSPC on Linux
let writer = FileWriter::new("/dev/full");
assert!(matches!(writer.append(&e).await, Err(Error::Io(_))));
```

`/dev/full` **does not exist on macOS**. The project's CI `test` job runs on `macos-latest` (the project has macOS-only FFI, so the runner cannot be swapped). The test would be skipped/error on the gating platform, leaving the production error-mapping branch effectively unexercised in CI — a silent coverage hole behind a green check.

A portable construction exists and was used in the fix: create a regular file inside a `tempdir`, then target a path *under* that file (`/audit.ndjson`) → `ENOTDIR` on both macOS and Linux → deterministic `Err(Io(_))`. The point is test-writer should have known to choose the portable form because it should have known the runner OS.

## Why existing issues don't cover it

- **#253** (research-agent OS-version advisory checks) and **#255** (dx-engineer OS-version preflight) are about *tool/dependency* OS-compat at setup time — not about *test code* targeting the wrong OS.
- **#330** (headless modality blindness) is about render/input modality, not OS-portability of a system-call-level test.
- **#225** (setup-env language assumption) is toolchain detection, not test portability.

No existing issue says: *test-writer must consult the CI runner matrix before emitting any OS/syscall-specific test and either use a portable construction or `#[cfg]`-gate it to the platforms CI actually runs.*

## Proposed fix

test-writer prompt addition:

> Before emitting any test that depends on platform-specific behavior — special device files (`/dev/full`, `/dev/null` semantics), syscall errno specifics, path separators, permission models, signal behavior — read the CI workflow(s) to determine the runner OS matrix. Either (a) use a construction that is deterministic on **every** runner OS in the matrix (preferred), or (b) `#[cfg(target_os = "…")]`-gate the test to the platforms CI runs and ensure at least one matrix entry exercises it. Never emit a test whose failure-injection mechanism is absent on the gating runner.

Optionally a deterministic lint (ties into #336): flag known-non-portable tokens (`/dev/full`, `/proc/`, `/sys/`, etc.) in test sources when the CI matrix lacks a matching `target_os`.

## Severity

P3 enhancement. First observed instance; cheap to fix; prevents a class of silent CI coverage holes on cross-platform or platform-pinned projects.

## Cross-references

- #253, #255 — OS-version awareness for research-agent / dx-engineer (setup-time analogues)
- #330 — modality blindness in headless test runs (sibling "green but blind" family)
- #336 — deterministic pre-review lint layer (home for the token detector)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.