BOHICA-LABS / BOHICA-LABS/vsdd-factory

Phase-4: evaluator dispatch should verify release binary is built from evaluated SHA (rebuild-fresh preflight)

Open
#525 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Symptom

Phase-4 holdout-evaluator dispatched at evaluated_sha `X` may execute a
`target/release/` (or equivalent) that was built from a PARENT of
`X`. When the fix at `X` changes runtime behavior of an audit / observable
surface, the evaluator observes pre-fix behavior and reports it as a
schema-drift or doc-drift false positive.

## Concrete instance

akey Phase-4 Pass-3 non-hardware half, 2026-07-05, evaluated_sha
`1454851bf986265b3c3116fb6364bcffdbc95a43`.

- Evaluated commit landed at `2026-07-05 16:47:41 -0500` — it is the fix
for a Pass-2 audit-content finding (extension name moved from
`.hardware_outcome.error` to `.decision.reason`).
- Release binary in the sandbox: `target/release/akey` dated
`2026-07-04 22:34` — ~18h older than the evaluated commit.
- Evaluator observed `.decision.reason == null` and extension name at
`.hardware_outcome.error`, reported as `F-HS002-EVAL-02`.
- Orchestrator ground-truth adjudicated: source at 1454851 matches the
doc; runtime doesn't match source because binary predates the fix.
- Product action: none. Pipeline action: this ticket.

Adjudication doc: `.factory/holdout-scenarios/evaluations/phase-4-pass-3-adjudication-addendum.md`

## Root cause

`phase-4-holdout-evaluation` workflow (see
`workflows/phases/phase-4-holdout-evaluation.lobster`) does not gate on:

1. \"Is there a release/debug artifact on disk for the evaluator to
execute?\"
2. \"If so, was it built from the evaluated SHA (or a descendant that
includes the evaluated SHA)?\"

The evaluator agent is spawned with a Bash+Read profile and is expected
to run the product binary end-to-end for hardware / audit / socket
scenarios. If the binary is stale, evaluator observations are stale.

## Proposed fix

**Preflight gate before evaluator dispatch:**

1. Locate the built artifact (project-config-driven, e.g.
`target/release/` for Rust, or a shipped Docker image tag).
2. Compare artifact mtime OR embedded build-time-SHA (if the project
embeds `git rev-parse HEAD` at build time) against the evaluated
commit's timestamp / SHA.
3. If artifact is missing or older than the evaluated commit — either:
- (a) Force-rebuild in-pipeline (`cargo build --release` or project's
`just build` / `make build`) BEFORE spawning evaluators, OR
- (b) Abort with a `preflight-failure: stale-binary` verdict and
surface the issue to the orchestrator for user-visible remediation.

Behavior should be configurable per project (build costs vary widely). A
sensible default is (a) with a max-build-time budget, falling back to
(b).

## Companion ticket

Related: #518 (Phase-4 preflight scenario-prereqs) — the two are natural
siblings and could share a preflight harness. #518 verifies that
scenario-required state exists; this verifies that the artifact
representing the SUT is fresh.

## Severity

**P2 methodology gap.** Silently invites false-positive product findings
that consume orchestrator ground-truth-adjudication cycles per instance.
Once caught (this session), cost is bounded to source-reading against
the evaluated SHA — but time-lost is real, and human operator running
Pass-3 by hand from Recipe B could plausibly be misled by a stale-binary
observation and file a phantom product bug on `akey` (or equivalent).

## Cross-references

- akey Phase-4 Pass-3 adjudication addendum:
`.factory/holdout-scenarios/evaluations/phase-4-pass-3-adjudication-addendum.md`
- drbothen/vsdd-factory#465 (evaluator hallucination class — related
but distinct: #465 covers 'evaluator invents facts'; this covers
'evaluator reports true facts about a stale build')
- drbothen/vsdd-factory#517 (Drift Items verification gate)
- drbothen/vsdd-factory#518 (Phase-4 preflight scenario-prereqs)
- drbothen/vsdd-factory#474 (family-independence gate)
- drbothen/vsdd-factory#516 (research-agent MCP provisioning)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.