BOHICA-LABS / BOHICA-LABS/vsdd-factory
Proposal: per-agent model rationale, dispatch outcome telemetry, escalation ladders, uniform effort hints
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
We mined all three of our running vsdd-factory rc.21 projects (akey — CLI tool; ftc-blue — Godot game; switchboard-blue — Go daemon): **4,602 factory dispatches extracted from session transcripts + dispatcher logs (95.8–98.5% join), 169 bad-outcome chains, every chain hand-classified against its own factory's `.factory/` evidence before anything counted against a model.** This issue proposes four engine changes the evidence supports. (Our analysis repo is private; happy to share the full reports/method on request.)
## The result that motivates everything
| corpus | chains | spec_defect | process_gap | prompt_defect | model_capability | by_design |
|---|--:|--:|--:|--:|--:|--:|
| akey | 63 | 20 | 21 | 0 | 16 | 6 |
| ftc-blue | 70 | 11 | 25 | 0 | 2 | 32 |
| switchboard-blue | 36 | 12 | 13 | 0 | 1 | 10 |
| **fleet** | **169** | **43** | **59** | **0** | **19** | **48** |
**84% of real (non-by_design) bad outcomes are not model capability.** All 19 capability chains fleet-wide are one cell (sonnet on code_write). The factories with mature process registries (lessons.md, known-issues.md) drove capability share from 28% (akey, the youngest) to 4–5% — process feedback, not model upgrades, is the first-order quality lever. Corroborating: implementer first-pass acceptance spans 42% / 66% / 82% across the three factories on the same engine and same model — the spread is spec maturity and process feedback, not the model.
## Ask 1 — `model_rationale:` frontmatter per agent definition
rc.21 pins 7 roles to opus and ~27 to sonnet with no recorded reasoning. Auditing any pin today means reconstructing intent from nothing. One sentence per agent definition makes pins auditable and revisable.
## Ask 2 — per-dispatch outcome telemetry
At dispatch completion, log one record — `(role, model, phase, story, accepted_or_respawned)` — into `.factory/logs/`, with `downstream_finding_count` backfilled when the next stage runs. Why retrospective mining can't substitute: two-label outcome coverage fails its floor **structurally on 3/3 factories** (28.6% / 34.3% / 35.4%) for identical reasons — holdout gates pending, and the PR-comment join is degenerate because factory-converged PRs merge with zero human comments. It's also the only way to detect prompt defects (0/169 retrospectively — either the rc.21 templates are mature or the class is retrospectively undetectable; emission disambiguates).
## Ask 3 — escalation ladders, both directions
**Upward:** retry a failed dispatch one tier up before re-prompting. akey STORY-014's seven consecutive partial-fix regressions (your #440/#441) shrink to ~two with telemetry-detected respawns + a ladder. Every ladder retry is also a free paired observation — the corpus currently contains zero (0/4,602 dispatches overrode a pin), so nothing about the current pins is empirically testable.
**Downward:** adversary CLEAN/NITPICK rate by convergence pass bucket, on independent factories:
| corpus | p1–3 | p4–6 | p7+ |
|---|--:|--:|--:|
| ftc-blue | 80% | 88% | 97% |
| switchboard-blue | 80% | 88% | 95% |
High-pass re-reviews are overwhelmingly confirmatory, yet run full-strength opus — and adversary is 33% of fleet dispatches. Full effort on passes 1–3, reduced after, is the clearest cost lever in the data.
## Ask 4 — uniform per-dispatch effort hints
The effort knob is untouched fleet-wide; the downward ladder above is naturally expressed as effort reduction on late confirmatory passes. Supporting it uniformly makes ask 3's downward direction expressible.
## Tuning candidates from the data (evidence, not demands)
- state-manager × mechanical table edits is haiku-shaped: n=547 fleet-wide, ~1.1k mean tokens in, zero capability attributions on any factory.
- Adversary late-pass effort-down per the table above.
Per-model context-window/effort simulation results (same methodology) are in progress and can follow in this thread.
Contributor guide
Assessment
This issue has not been assessed yet.