End-to-end workflow tests spanning sandbox + agent harness + environment

Summary

Our nightly CI runs end-to-end tests on GPU and CPU, but tests components in isolation. A recent attempt to run a canonical user workflow (NeMo Gym eval, Open Shell sandbox provider, Hermes agent harness, workplace assistant tasks) hit four independent failures, none of which were caught by existing tests. We need broader workflow tests that exercise a complete "user job to be done" across sandbox provider, agent server, tool delivery, and verification.

Motivation

Every individual component passed its own tests. The failures were all in the seams between components. Benchmark-specific coverage (for example SWE-bench with one hard-coded provider) does not represent the combinations real users run, so integration regressions ship undetected and surface during demos and customer engagements.

Failures observed

Each of these should become a regression test.

1. No agent server bridges sandbox infra to Hermes. Sandbox providers exist but do not connect to available agent harnesses out of the box. Running the workflow required an unmerged draft PR containing a generic sandbox agent server. This is a gap rather than a bug, but it means the advertised path does not work end to end.

2. Sandbox name exceeds Open Shell gateway limit. Generated sandbox names exceed Open Shell's 19-character maximum, so every container instance creation fails immediately. Provider-side name generation does not honor provider constraints.

3. MCP tool reception disabled in the Hermes agent server. Workplace assistant defines its tools in a resource server delivered over MCP. Our Hermes agent server implementation has that functionality disabled, so the agent never receives the tools.

4. Verification logic hard-coded to one harness's tool-name mutation. Harnesses mutate tool names differently when passing them to the model. Our verification parses the mutated name using Claude Code's convention, producing false failures for Hermes and any other harness. A fix is under discussion: track MCP provenance as a sidecar so the original server and tool names are recoverable.

Proposed scope

Define a small matrix of end-to-end workflow tests that assert a complete run succeeds, not just that individual pieces respond.

Each test should run a real task to completion and assert on the final verification result, so provider constraints, tool delivery, and name parsing are all exercised.

Acceptance criteria

• At least one nightly test runs a full workflow across sandbox provider, agent harness, MCP tool delivery, and verification, and fails if any seam breaks
• Regression tests exist for each of the four failures above
• Provider-specific constraints such as name length limits are asserted rather than discovered at runtime
• Verification passes identically across at least two harnesses for the same task, confirming no harness-specific parsing dependency
• Test matrix is documented so adding a new provider or harness has a clear checklist

Open questions

• How much of the current manual setup (Open Shell on a local machine) can be automated in CI, and what needs a hosted deployment?
• Do we need an Open Shell cluster or deployment to include it in the nightly matrix, and who owns provisioning it?
• Which task should serve as the canonical smoke workflow, given runtime cost constraints on nightly CI?