BOHICA-LABS / BOHICA-LABS/vsdd-factory

bug(dispatch): agent model header may not reflect the dispatched model — undermines BC-5.39.001 model-diversity guarantee

Open
#468 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Context

BC-5.39.001 requires model diversity across adversarial passes for per-story convergence. Orchestrators rotate the `model` parameter on Agent tool calls (e.g., `subagent_type: vsdd-factory:adversary`, `model: opus` vs `model: sonnet`) to obtain fresh-perspective reviews.

Observed on akey STORY-015 convergence arc (this thread):

- **Pass 6:** dispatched with `model: opus`. The adversary's own report header self-identified as `"Model: sonnet"` / "Sonnet, fresh". Verification substance was grounded (file:line citations, red-gate correct), but the model identity is ambiguous.
- **Pass 7:** dispatched with `model: opus`. The adversary's report header self-identified as `"Model: opus"`. Explicit and consistent.
- **Pass 8:** dispatched with `model: sonnet`. Header: `"Model: claude-sonnet-4-6"`.

The pass 6 discrepancy has two candidate explanations:

**(a)** The `model: opus` override at Agent-tool dispatch level did NOT take effect (agent used the default / inherited model, which happened to be sonnet). If true: model-rotation for BC-5.39.001 diversity is fictional whenever an orchestrator asks for a non-default model but the harness silently ignores or falls-through.

**(b)** The model override DID take effect (opus ran), but the agent's own system prompt or preamble caused it to introspect wrong and self-report "Sonnet" in the header. If true: the header self-identification is unreliable evidence and orchestrators need a different signal to verify rotation.

Either explanation undermines the auditability of BC-5.39.001's diversity claim.

## Proposal

1. **Instrument the Agent tool / factory harness to emit the actual model used** in the task-completion notification metadata (already visible to the orchestrator as `usage.subagent_tokens` etc. — add `usage.model_id`).
2. **Update the adversary agent prompt to fetch its actual model identity from a source-of-truth** (env var, system-injected identity), not self-introspection.
3. **Add a preflight in per-story convergence:** if the orchestrator dispatched with `model: X`, verify the completion reported `model_id: X`. Log a warning if not — the convergence pass counter should NOT decrement, but the model-diversity accounting for this story must reflect what actually ran.

## Reproduction / Evidence

Session on ArcavenAE/akey `story/STORY-015-hot-reload`, three adversary passes 6/7/8 dispatched with different model params. Pass 6 headers "Sonnet" despite `model: opus` in the Agent tool call. Passes 7 and 8 self-identify consistently. No public artifact — this is factory-internal.

## Related

- #465 (adversary hallucination — indirectly relevant since header self-report unreliability is a form of self-reporting drift)
- BC-5.39.001 model-diversity clause

## Suggested labels

`bug`, `adversary`, `dispatch`, `observability`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.