BOHICA-LABS / BOHICA-LABS/vsdd-factory

observation(orchestrator): consistency-validator + spec-reviewer verdict gap on shared input suggests prompt calibration drift

Open
#278 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

In Wave 3 pre-implementation review on switchboard-blue, both `consistency-validator` and `spec-reviewer` were dispatched in parallel on the same set of Wave 3 specs (4 stories + 7 BCs + 2 ADRs + 1 VP + error-taxonomy update).

**consistency-validator verdict:** `PASS_WITH_OBSERVATIONS` — 0 CRITICAL / 3 HIGH / 4 MEDIUM / 2 LOW / 4 OBSERVATION.

**spec-reviewer verdict:** `REVISE_BEFORE_SHIP` — 3 CRITICAL / 8 HIGH / 8 MEDIUM / 6 LOW / 4 COMMENDATION.

The two agents found a similar baseline of issues but classified them at very different severities. spec-reviewer's CRITICALs (VP-058 proof harness compile failure, ADR-009 internal contradictions) were genuinely critical — they would have caused mid-story rework if ignored. consistency-validator either missed them (VP-058 compile fail) or downgraded them (ADR-009 signature mismatch as MEDIUM instead of CRITICAL).

## Why this matters

The verdict gap (PASS vs REVISE) on shared input means orchestrator gating depends on WHICH agent is dispatched. If only consistency-validator had run, Wave 3 would have launched without the 3 CRITICALs being addressed.

Possible causes:
1. **Prompt-defined severity scales differ.** consistency-validator's "HIGH" may equal spec-reviewer's "MEDIUM" in practice.
2. **Different scope.** consistency-validator focuses on cross-file traceability; spec-reviewer focuses on testability and internal logical consistency. Different scopes naturally surface different defects.
3. **Cognitive-diversity bonus.** spec-reviewer is documented as "different model family"; that diversity may inherently surface defects consistency-validator's family misses.

(2) and (3) are positive features; (1) is a calibration bug.

## Mitigation suggestions

- Audit the severity definitions in both agent system prompts.
- Define a shared severity rubric (CRITICAL = blocks merge, HIGH = blocks current wave, MEDIUM = blocks story, LOW = doc/style, OBSERVATION = informational).
- Consider making spec-reviewer's CRITICAL findings a HARD gate (auto-block) regardless of consistency-validator's verdict.

## Reproduction

Switchboard-blue session `b8690230-7b5d-4e57-b71c-8b9ffeee3d9a`. Wave 3 pre-impl burst. Reports at:
- `.factory/cycles/cycle-1/wave-3/pre-impl-consistency-report.md`
- `.factory/cycles/cycle-1/wave-3/pre-impl-spec-review.md`

## Related (this batch)

- drbothen/vsdd-factory#272 — P19: architect hallucinated internal/ packages
- drbothen/vsdd-factory#273 — P20: parallel agent burst commit co-mingling
- drbothen/vsdd-factory#274 — P21: ADR amendment + code update drift across revisions
- drbothen/vsdd-factory#275 — P22: parallel agents make contradictory architectural decisions
- drbothen/vsdd-factory#276 — P23: spec-reviewer cross-eye should be mandatory post-spec-burst
- drbothen/vsdd-factory#277 — P24: derived summary fields go stale on primary content add

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.