BOHICA-LABS / BOHICA-LABS/vsdd-factory
Adversary methodology: detect tokio-worker starvation from blocking I/O in #[tokio::test] integration tests
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
Ten adversarial passes on STORY-017 (akey greenfield pipeline) failed to detect a test-scaffolding defect where a `#[tokio::test]` integration test starves its single tokio worker by calling a blocking OS API (`std::process::Child::wait()`) while requiring the same worker to make async progress on a spawned `daemon_task`. The defect passed all 10 CLEAN adversarial passes locally, but produced deterministic CI failure on GitHub Actions macos-latest.
## Observed instance
- Story: STORY-017 (akey — LOCAL_PEERPID peer PID capture on macOS)
- Test: AC-A1 integration test — `caller_pid = 0` in daemon audit log
- CI run: GHA run 28727397489 (5 consecutive failures on macos-latest)
- Local result: green on identical macOS 26 arm64 hardware
- Adversarial audit: Pass 1..Pass 10 all CLEAN; scaffolding gap not surfaced
## Root cause
`#[tokio::test]` defaults to `flavor = "current_thread"` (one worker). The test:
1. Spawns `daemon_task = tokio::spawn(daemon_start(...))`.
2. Polls for socket-bind readiness.
3. Spawns a subprocess (`pid_client`) via `std::process::Command`.
4. Calls `subprocess.wait()` **synchronously** on the tokio worker thread.
Step 4 starves the tokio worker. `daemon_task` cannot progress from `bind()` to `socket.run()` — the point where the accept OS thread is spawned. All connections queue in the kernel backlog. `pid_client`'s 5s read timeout fires (daemon never responds because no accept thread exists yet), `pid_client` drops its socket and exits. Only then does `subprocess.wait()` return and the tokio worker resume. Daemon reaches `socket.run()`, spawns accept thread, accept thread rapidly drains the backlog — but both peers are already dead. `getsockopt(LOCAL_PEERPID)` returns `ret=-1 errno=ENOTCONN` for every backlogged fd. Audit log shows `caller_pid=0` because the peer-PID lookup failed.
Locally passes because the scheduler happens to interleave the tokio worker before `wait()` blocks it. The CI VM's scheduler serializes differently and always loses the race.
Timing evidence: the 5s `pid_client` read timeout aligns exactly with the ENOTCONN cluster in the daemon accept log.
## Fix applied on the product side
Wrap the blocking `subprocess.wait()` in `tokio::task::spawn_blocking` so the tokio worker remains free to drive `daemon_task`. Alternatively, switch the test to `#[tokio::test(flavor = \"multi_thread\")]`.
## Proposed engine rule
For any `#[tokio::test]`, adversary should flag test bodies that call **blocking OS APIs after spawning an in-runtime background task with `tokio::spawn`**. This is a systemic pattern: any such test has a latent worker-starvation bug that only surfaces under adverse scheduler conditions (CI VMs, loaded runners).
### Negative pattern (must flag)
```rust
#[tokio::test] // defaults to current_thread — one worker
async fn some_test() {
let daemon_task = tokio::spawn(daemon_start(...)); // needs worker
wait_for_socket_bind().await;
let mut child = std::process::Command::new(\"pid_client\").spawn().unwrap();
let status = child.wait().unwrap(); // BLOCKS the only worker
// daemon_task cannot progress until wait() returns
}
```
### Blocking APIs to check for (non-exhaustive)
- `std::process::Child::wait`, `wait_with_output`, `try_wait` (spin loops)
- `std::io::BufRead::read_line`, `Read::read`, `Read::read_to_end` on a sync handle
- `std::thread::sleep`
- `std::sync::Mutex::lock` / `RwLock::{read,write}` on a contended lock
- `std::sync::mpsc::Receiver::recv`
- Any FFI call marked blocking (e.g. `libc::read`, `getaddrinfo`, sync file I/O)
### Recommended remediations (adversary should suggest one)
1. Wrap the blocking call in `tokio::task::spawn_blocking(...)` and `.await` it.
2. Switch the attribute to `#[tokio::test(flavor = \"multi_thread\", worker_threads = 2)]`.
3. Replace the blocking API with its tokio-async equivalent (`tokio::process::Command`, `tokio::io::AsyncReadExt`, `tokio::time::sleep`, `tokio::sync::Mutex`).
## Cross-references
Precedents on adversary-methodology gaps discovered in the akey pipeline:
- #440
- #441
- #466
- #467
- #468
Contributor guide
Assessment
This issue has not been assessed yet.