BOHICA-LABS / BOHICA-LABS/vsdd-factory
bug(dispatch): agent model header may not reflect the dispatched model — undermines BC-5.39.001 model-diversity guarantee
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Context
BC-5.39.001 requires model diversity across adversarial passes for per-story convergence. Orchestrators rotate the `model` parameter on Agent tool calls (e.g., `subagent_type: vsdd-factory:adversary`, `model: opus` vs `model: sonnet`) to obtain fresh-perspective reviews.
Observed on akey STORY-015 convergence arc (this thread):
- **Pass 6:** dispatched with `model: opus`. The adversary's own report header self-identified as `"Model: sonnet"` / "Sonnet, fresh". Verification substance was grounded (file:line citations, red-gate correct), but the model identity is ambiguous.
- **Pass 7:** dispatched with `model: opus`. The adversary's report header self-identified as `"Model: opus"`. Explicit and consistent.
- **Pass 8:** dispatched with `model: sonnet`. Header: `"Model: claude-sonnet-4-6"`.
The pass 6 discrepancy has two candidate explanations:
**(a)** The `model: opus` override at Agent-tool dispatch level did NOT take effect (agent used the default / inherited model, which happened to be sonnet). If true: model-rotation for BC-5.39.001 diversity is fictional whenever an orchestrator asks for a non-default model but the harness silently ignores or falls-through.
**(b)** The model override DID take effect (opus ran), but the agent's own system prompt or preamble caused it to introspect wrong and self-report "Sonnet" in the header. If true: the header self-identification is unreliable evidence and orchestrators need a different signal to verify rotation.
Either explanation undermines the auditability of BC-5.39.001's diversity claim.
## Proposal
1. **Instrument the Agent tool / factory harness to emit the actual model used** in the task-completion notification metadata (already visible to the orchestrator as `usage.subagent_tokens` etc. — add `usage.model_id`).
2. **Update the adversary agent prompt to fetch its actual model identity from a source-of-truth** (env var, system-injected identity), not self-introspection.
3. **Add a preflight in per-story convergence:** if the orchestrator dispatched with `model: X`, verify the completion reported `model_id: X`. Log a warning if not — the convergence pass counter should NOT decrement, but the model-diversity accounting for this story must reflect what actually ran.
## Reproduction / Evidence
Session on ArcavenAE/akey `story/STORY-015-hot-reload`, three adversary passes 6/7/8 dispatched with different model params. Pass 6 headers "Sonnet" despite `model: opus` in the Agent tool call. Passes 7 and 8 self-identify consistently. No public artifact — this is factory-internal.
## Related
- #465 (adversary hallucination — indirectly relevant since header self-report unreliability is a form of self-reporting drift)
- BC-5.39.001 model-diversity clause
## Suggested labels
`bug`, `adversary`, `dispatch`, `observability`
Contributor guide
Assessment
This issue has not been assessed yet.