BOHICA-LABS / BOHICA-LABS/vsdd-factory

Proposal: per-agent model rationale, dispatch outcome telemetry, escalation ladders, uniform effort hints

Open
#463 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

We mined all three of our running vsdd-factory rc.21 projects (akey — CLI tool; ftc-blue — Godot game; switchboard-blue — Go daemon): **4,602 factory dispatches extracted from session transcripts + dispatcher logs (95.8–98.5% join), 169 bad-outcome chains, every chain hand-classified against its own factory's `.factory/` evidence before anything counted against a model.** This issue proposes four engine changes the evidence supports. (Our analysis repo is private; happy to share the full reports/method on request.)

## The result that motivates everything

| corpus | chains | spec_defect | process_gap | prompt_defect | model_capability | by_design |
|---|--:|--:|--:|--:|--:|--:|
| akey | 63 | 20 | 21 | 0 | 16 | 6 |
| ftc-blue | 70 | 11 | 25 | 0 | 2 | 32 |
| switchboard-blue | 36 | 12 | 13 | 0 | 1 | 10 |
| **fleet** | **169** | **43** | **59** | **0** | **19** | **48** |

**84% of real (non-by_design) bad outcomes are not model capability.** All 19 capability chains fleet-wide are one cell (sonnet on code_write). The factories with mature process registries (lessons.md, known-issues.md) drove capability share from 28% (akey, the youngest) to 4–5% — process feedback, not model upgrades, is the first-order quality lever. Corroborating: implementer first-pass acceptance spans 42% / 66% / 82% across the three factories on the same engine and same model — the spread is spec maturity and process feedback, not the model.

## Ask 1 — `model_rationale:` frontmatter per agent definition

rc.21 pins 7 roles to opus and ~27 to sonnet with no recorded reasoning. Auditing any pin today means reconstructing intent from nothing. One sentence per agent definition makes pins auditable and revisable.

## Ask 2 — per-dispatch outcome telemetry

At dispatch completion, log one record — `(role, model, phase, story, accepted_or_respawned)` — into `.factory/logs/`, with `downstream_finding_count` backfilled when the next stage runs. Why retrospective mining can't substitute: two-label outcome coverage fails its floor **structurally on 3/3 factories** (28.6% / 34.3% / 35.4%) for identical reasons — holdout gates pending, and the PR-comment join is degenerate because factory-converged PRs merge with zero human comments. It's also the only way to detect prompt defects (0/169 retrospectively — either the rc.21 templates are mature or the class is retrospectively undetectable; emission disambiguates).

## Ask 3 — escalation ladders, both directions

**Upward:** retry a failed dispatch one tier up before re-prompting. akey STORY-014's seven consecutive partial-fix regressions (your #440/#441) shrink to ~two with telemetry-detected respawns + a ladder. Every ladder retry is also a free paired observation — the corpus currently contains zero (0/4,602 dispatches overrode a pin), so nothing about the current pins is empirically testable.

**Downward:** adversary CLEAN/NITPICK rate by convergence pass bucket, on independent factories:

| corpus | p1–3 | p4–6 | p7+ |
|---|--:|--:|--:|
| ftc-blue | 80% | 88% | 97% |
| switchboard-blue | 80% | 88% | 95% |

High-pass re-reviews are overwhelmingly confirmatory, yet run full-strength opus — and adversary is 33% of fleet dispatches. Full effort on passes 1–3, reduced after, is the clearest cost lever in the data.

## Ask 4 — uniform per-dispatch effort hints

The effort knob is untouched fleet-wide; the downward ladder above is naturally expressed as effort reduction on late confirmatory passes. Supporting it uniformly makes ask 3's downward direction expressible.

## Tuning candidates from the data (evidence, not demands)

- state-manager × mechanical table edits is haiku-shaped: n=547 fleet-wide, ~1.1k mean tokens in, zero capability attributions on any factory.
- Adversary late-pass effort-down per the table above.

Per-model context-window/effort simulation results (same methodology) are in progress and can follow in this thread.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.