[P0] Run skill-enabled coding agents inside GPU-backed sandboxes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
Use cases, pain points, and background
Some skill evaluations require the agent to execute real hardware-accelerated workflows. Gym can load skills into supported coding agents, and its sandbox API can request GPU resources, but the skill-enabled coding-agent path does not currently provide an end-to-end way to run the agent itself inside a GPU-backed sandbox. This prevents execution-based evaluation and can make a text-only response appear successful even when no workload ran.
Description:
Add a provider-neutral execution path that runs supported skill-enabled coding agents inside a Gym-managed sandbox with GPU resources.
The path should:
- create one isolated sandbox per rollout using the existing sandbox provider abstraction;
- support image, working directory, timeout, environment, and GPU resource configuration;
- stage the selected skills into the agent runtime's native discovery location inside the sandbox;
- launch the coding agent inside that sandbox rather than on the Gym host;
- preserve model-server and resources-server connectivity without exposing credentials in logs or result artifacts;
- collect the response, trajectory, usage, command outcomes, and relevant artifacts; and
- reliably clean up the sandbox after success, failure, or cancellation.
Design:
Prefer extending the reusable external-harness/coding-agent integration over adding a benchmark-specific runner. Reuse nemo_gym.sandbox, SandboxSpec, and SandboxResources. Keep skill staging request-scoped so concurrent rollouts cannot share or overwrite skill state.
This is related to #2082, which covers reusable sandbox ownership and sharing primitives. This issue is specifically the end-to-end integration for skill-enabled coding agents executing in a GPU-backed sandbox.
Out of scope:
- Provisioning or operating a GPU cluster.
- Benchmark-specific container images or test data.
- A new sandbox provider.
- Changing how the agent runtime chooses which staged skill to activate.
Acceptance Criteria:
- A supported coding agent can run entirely inside a configured sandbox with
resources.gpuenabled. - The same rollout can load a selected skill set and execute a GPU probe or workload from inside the sandbox.
- A no-skill baseline can use the identical image, resources, task input, and agent configuration.
- Rollout output records sandbox execution evidence and skill provenance without leaking secrets.
- Concurrent rollouts are isolated from one another.
- Sandboxes are cleaned up on success, timeout, cancellation, and agent failure.
- Unit tests cover configuration, skill staging, launch, collection, and cleanup; an integration test covers a GPU-backed execution path.
- User documentation includes a minimal provider-neutral example.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the reusable external-harness/coding-agent integration and reading nemo_gym.sandbox, SandboxSpec, and SandboxResources. Use the existing unit-test structure for configuration, staging, launch, collection, and cleanup, then add the requested GPU-backed integration test and provider-neutral documentation example; done means all listed acceptance criteria pass without credential leakage or cross-rollout sharing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100