BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(adversarial-refinement): race-finding fixes that install sync barriers must fail closed and be stress-validated under constrained schedulers

Open
#502 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Context

Project: switchboard-blue (Go, greenfield cycle-1, currently post-SHIPPED steady state, plugin 1.0.0-rc.21).

A metadata-only PR (`.run.yaml`, zero code changes) failed the Quality Gate **twice in a row** on two *different* pre-existing flaky tests in the same package — both scheduling races that shared `-race` CI runners widen. Both tests were produced/adjusted by factory-dispatched agents during Phase 3/5.

## The pattern

**Instance 1 — fail-open sync barrier.** A Pass-3 L2 adversarial race finding correctly flagged a `time.Sleep`+`Gosched` synchronization in a test (register-before-serve invariant, verifying an atomic flag set at Serve entry). The dispatched fix replaced it with a dial-until-accept loop — the right idea — but:

1. The dial loop fast-fails on connection-refused (microseconds per attempt), so all 50 attempts can complete before the Serve goroutine is ever scheduled. No sleep between attempts.
2. When the barrier failed to establish, the test **logged and asserted anyway** (`t.Log("...may be flaky")` then proceed), producing a false invariant-violation failure instead of a loud barrier failure.

The fix for a race finding introduced a new race with a fail-open observability hole.

**Instance 2 — success-path race in a shutdown-drop test.** A Ruling-compliance test (connection accepted post-Shutdown must be dropped) reads from the client half of a `net.Pipe` with a deadline. If the server drops the connection **fast enough** — i.e., the exact behavior the test verifies — `SetReadDeadline` returns `io.ErrClosedPipe` and the test `t.Fatal`'d on its own PASS condition.

## Why it slipped through (introspection)

- Both tests are green on fast dev machines; the windows only open under `-race` + shared-runner scheduler starvation.
- The adversary reviews test *logic* vs spec, but no pass exercises tests under scheduler pressure. A race-finding *fix* is accepted if it removes the flagged primitive (`time.Sleep`), not if the replacement barrier is proven sound.
- No convergence-dimension check catches "test fails on its own success condition" (instance 2) — that requires reasoning about the timing of the behavior under test relative to the observation channel.

## Proposed improvements

1. **Fail-closed rule for sync barriers in dispatched race fixes:** when test-writer/implementer installs a synchronization barrier in a test (dial loop, channel wait, condition poll), the barrier must (a) yield between attempts, and (b) `t.Fatal` with a barrier-specific message if it never establishes — never log-and-assert-anyway. Candidate for the test-writer agent prompt and the adversarial race-finding fix-routing instructions.
2. **Stress validation gate for race-finding fixes:** the fix-PR evidence for any race/flake finding should include a constrained-scheduler stress run (e.g. `GOMAXPROCS=2 go test -race -count=10 -run `), not just a single green run.
3. **Adversarial axis candidate:** "observation-channel timing" — for tests that observe an asynchronous behavior through a channel/pipe/socket, ask whether the observed behavior *completing quickly* can break the observation mechanism itself (instance 2's class).

## Evidence

- Quality Gate run 28732071943: attempt 1 failed `TestRegister_AfterServeReturnsError`, attempt 2 failed `TestServe_DrainCompletesWithinBudget_RulingI`, attempt 3 green.
- Fix PR (test-only, both instances): ArcavenAE/switchboard-blue#84.

Severity: MEDIUM (process-gap — factory-authored race fixes can institutionalize new flakes; erodes trust in the Quality Gate signal and burns rerun cycles on every downstream PR).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.