enhancement(holdout-evaluator): evaluability constraints — hardware/human-gated criteria need a first-class category and a resume-queue

## Summary

The holdout-evaluator SKILL and the Phase-4 gate arithmetic assume every acceptance criterion is machine-observable by an agent with shell + build tools. For products that legitimately depend on **physical hardware** (FIDO2 tokens, robotics, IoT sensors, audio interfaces, GPUs with specific firmware) or on **human participation** (focus-theft detection, ergonomics observation, timing-under-load subjective ratings), a large fraction of holdout scenarios are unevaluable without those inputs.

The current pipeline has:
- No first-class category for \"unevaluable by this evaluator\" — evaluators have to invent one ad-hoc.
- No protocol for how such criteria contribute to (or are excluded from) the satisfaction denominator.
- No resume-queue for outstanding human/hardware verifications — the orchestrator has nowhere to track \"waiting on human to run HS-003 with YubiKey\" between Phase-4 and Phase-5.

## Evidence — Phase-4 evaluation of akey, 2026-07-03

Product is a FIDO2-hardware SSH agent. Seven holdout scenarios; six require physical hardware or a human participant:

| HS | Hardware/human requirement |
|----|----------------------------|
| HS-001 (100 signing requests) | FIDO2 sk-key + git |
| HS-002 (LLM bypass prevention) | fully evaluable — no hardware needed |
| HS-003 (hardware-key mode) | YubiKey device |
| HS-004 (non-interruptive attention) | human participant + windowed terminal |
| HS-005 (audit-log durability) | FIDO2 sk-key (fail-closed only observable during signing) |
| HS-006 (known-good corpus, 25 signed commits) | FIDO2 sk-key + git |
| HS-007 (known-problematic corpus, container origins etc.) | FIDO2 sk-key + containers |

The evaluator handled it well by inventing a category on the fly (\"UNEVALUABLE-without-hardware\") and excluding those criteria from the denominator. Report [HERE — akey private repo, gate FAIL surfaced HS-002 Attack 3]. But two failure modes lurk in the ad-hoc handling:

1. **False-FAIL risk.** A less-cautious evaluator could score unevaluable criteria 0.0 (\"couldn't verify → didn't pass\"), tanking the mean satisfaction and forcing a gate FAIL that doesn't reflect the product's actual quality.
2. **Silent-truncation risk.** Alternatively an evaluator could quietly drop unevaluable criteria without reporting them at all — the gate then passes on a partial evaluation that is misrepresented as complete.

Additionally: the six unevaluable scenarios in this session have no scheduled resume. The orchestrator surfaced them to the human as a list in the review report, but nothing in `.factory/` tracks them as pending human-verification obligations that must clear before Phase-5 adversarial or Phase-7 convergence.

## Proposed shape

1. **First-class evaluability tag on every HS scenario.** Add a frontmatter field, e.g. `evaluability: [machine | hardware:FIDO2 | hardware:usb-serial | human:participant | human:security-review]`, produced by product-owner during Phase-1c holdout-scenario writing.
2. **Holdout-evaluator SKILL updates.**
- Read `evaluability` before scoring. For non-`machine` scenarios: evaluate every machine-observable sub-criterion inside the scenario (audit surfaces, error paths, CLI signatures, timeout knobs) and score satisfaction across evaluated criteria only. Explicitly mark the hardware/human-gated criteria as `pending-human-verification` — not scored 0, not scored 1.
- Emit a structured `human-verification-queue.md` output alongside the evaluation report, one entry per pending criterion, with the exact steps a human should run.
3. **Gate arithmetic clarification.** State whether the Phase-4 gate can PASS on a partial evaluation (with all machine-observable criteria passing) with the human-queue as a hard dependency for Phase-5 entry, or whether Phase-4 must genuinely block until every criterion is verified. My guess is the former with the queue as a Phase-5 prerequisite — but the doctrine is not written down.
4. **Resume-queue tracking.** `human-verification-queue.md` becomes a first-class artifact in `.factory/holdout-scenarios/evaluations/` with per-criterion status (pending / verified-pass / verified-fail / accepted-risk-with-signoff). Phase-5 entry checks the queue is empty (or documents accepted-risk overrides).

## Why file

Hardware-adjacent greenfield products are not a special case — as agents build progressively more physically-integrated systems (device drivers, robotics, embedded, biotech instrumentation, ICS), this becomes the default shape, not the exception. Ad-hoc evaluator handling is fine once; codifying it prevents both the false-FAIL and silent-truncation failure modes from ever landing.

## Non-goals for this issue

- Not proposing that agents get hardware access. The point is that the framework should recognize when it *can't* evaluate something and route to the human cleanly.
- Not addressing test-writer or story-writer scope for hardware-gated tests — separate concern (Phase-3, unit + integration; this issue is Phase-4-specific).