Milestone 4: standalone Hermes and SWE-bench Pro validation

Parallel work

Hermes

• Complete the agent server checklist.
• Use runtime direct SandboxAccess.
• Remove duplicate borrowed-provider configuration.
• Preserve Hermes observations and rollout-prefixed model/token capture.

SWE-bench Pro

• Complete the resources server checklist.
• Retain private task and verifier data at seed.
• Return direct SandboxAccess.
• Preserve patch extraction and fresh verifier sandboxes.

Single-agent processor

• Seed resources and agent sessions.
• Activate Hermes through rollout-prefixed /v1/responses.
• Close Hermes before verification.
• Verify and close resources.
• Preserve separate cookie jars and current rollout result projection.

Validation

• Run successful rollout and failure injection.
• Inspect observations, token/model capture, verifier fields, masking, retries, sidecars, and cleanup.

Required model set

Use these models for benchmark validation after their authoritative endpoint IDs are confirmed:

• Qwen 3.8 Flash 120B A6B 51B ngram embedding.
• Nemotron 3 Ultra.
• Qwen3.5-122B-A10B.
• DeepSeek-V4-Flash-0731.

Definition of done

• Standalone Gym produces valid SWE-bench Pro rollout results through the processor.
• Current and native paths preserve response, verifier, masking, retry, denominator, sidecar, observation, capture, and aggregation semantics.
• Model, verifier, timeout, cancellation, and abandoned-session failures complete bounded cleanup.
• Hermes cannot destroy the resources-owned sandbox.
• Generic collector and processor base contain no SWE-bench Pro logic.