BOHICA-LABS / BOHICA-LABS/vsdd-factory

Adversary methodology: detect tokio-worker starvation from blocking I/O in #[tokio::test] integration tests

Open
#490 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Ten adversarial passes on STORY-017 (akey greenfield pipeline) failed to detect a test-scaffolding defect where a `#[tokio::test]` integration test starves its single tokio worker by calling a blocking OS API (`std::process::Child::wait()`) while requiring the same worker to make async progress on a spawned `daemon_task`. The defect passed all 10 CLEAN adversarial passes locally, but produced deterministic CI failure on GitHub Actions macos-latest.

## Observed instance

- Story: STORY-017 (akey — LOCAL_PEERPID peer PID capture on macOS)
- Test: AC-A1 integration test — `caller_pid = 0` in daemon audit log
- CI run: GHA run 28727397489 (5 consecutive failures on macos-latest)
- Local result: green on identical macOS 26 arm64 hardware
- Adversarial audit: Pass 1..Pass 10 all CLEAN; scaffolding gap not surfaced

## Root cause

`#[tokio::test]` defaults to `flavor = "current_thread"` (one worker). The test:

1. Spawns `daemon_task = tokio::spawn(daemon_start(...))`.
2. Polls for socket-bind readiness.
3. Spawns a subprocess (`pid_client`) via `std::process::Command`.
4. Calls `subprocess.wait()` **synchronously** on the tokio worker thread.

Step 4 starves the tokio worker. `daemon_task` cannot progress from `bind()` to `socket.run()` — the point where the accept OS thread is spawned. All connections queue in the kernel backlog. `pid_client`'s 5s read timeout fires (daemon never responds because no accept thread exists yet), `pid_client` drops its socket and exits. Only then does `subprocess.wait()` return and the tokio worker resume. Daemon reaches `socket.run()`, spawns accept thread, accept thread rapidly drains the backlog — but both peers are already dead. `getsockopt(LOCAL_PEERPID)` returns `ret=-1 errno=ENOTCONN` for every backlogged fd. Audit log shows `caller_pid=0` because the peer-PID lookup failed.

Locally passes because the scheduler happens to interleave the tokio worker before `wait()` blocks it. The CI VM's scheduler serializes differently and always loses the race.

Timing evidence: the 5s `pid_client` read timeout aligns exactly with the ENOTCONN cluster in the daemon accept log.

## Fix applied on the product side

Wrap the blocking `subprocess.wait()` in `tokio::task::spawn_blocking` so the tokio worker remains free to drive `daemon_task`. Alternatively, switch the test to `#[tokio::test(flavor = \"multi_thread\")]`.

## Proposed engine rule

For any `#[tokio::test]`, adversary should flag test bodies that call **blocking OS APIs after spawning an in-runtime background task with `tokio::spawn`**. This is a systemic pattern: any such test has a latent worker-starvation bug that only surfaces under adverse scheduler conditions (CI VMs, loaded runners).

### Negative pattern (must flag)

```rust
#[tokio::test] // defaults to current_thread — one worker
async fn some_test() {
let daemon_task = tokio::spawn(daemon_start(...)); // needs worker
wait_for_socket_bind().await;
let mut child = std::process::Command::new(\"pid_client\").spawn().unwrap();
let status = child.wait().unwrap(); // BLOCKS the only worker
// daemon_task cannot progress until wait() returns
}
```

### Blocking APIs to check for (non-exhaustive)

- `std::process::Child::wait`, `wait_with_output`, `try_wait` (spin loops)
- `std::io::BufRead::read_line`, `Read::read`, `Read::read_to_end` on a sync handle
- `std::thread::sleep`
- `std::sync::Mutex::lock` / `RwLock::{read,write}` on a contended lock
- `std::sync::mpsc::Receiver::recv`
- Any FFI call marked blocking (e.g. `libc::read`, `getaddrinfo`, sync file I/O)

### Recommended remediations (adversary should suggest one)

1. Wrap the blocking call in `tokio::task::spawn_blocking(...)` and `.await` it.
2. Switch the attribute to `#[tokio::test(flavor = \"multi_thread\", worker_threads = 2)]`.
3. Replace the blocking API with its tokio-async equivalent (`tokio::process::Command`, `tokio::io::AsyncReadExt`, `tokio::time::sleep`, `tokio::sync::Mutex`).

## Cross-references

Precedents on adversary-methodology gaps discovered in the akey pipeline:

- #440
- #441
- #466
- #467
- #468

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.