githubnext / githubnext/gh-aw-test

Suggested test: copilot/docker-sbx-runtime

Closed
#6,735 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#45006 — Add `docker-sbx` runtime support to compiler

PR #45006 adds `docker-sbx` as a valid `sandbox.agent.runtime` value, running the agent inside a KVM-isolated Docker sbx microVM. This is a substantially different execution model from the existing `gvisor` and default runtimes, with no E2E test coverage.

## Proposed test

- **Workflow file**: `test-copilot-docker-sbx-runtime.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue`
- **Variant**: standard (nosandbox would defeat the purpose)

## Minimal test prompt sketch

Using `sandbox.agent.runtime: docker-sbx` (and `sudo: true`) in frontmatter, instruct the agent to create a GitHub issue with a fixed title confirming it ran inside the docker-sbx microVM. This validates the KVM pre-flight, sbx install, daemon startup, Docker Hub auth, smoke test, and agent execution end-to-end.

## New fixtures or secrets needed

Requires two new repository secrets: `DOCKER_PAT` and `DOCKER_USERNAME` (Docker Hub credentials used by the compiler-generated `docker login` step). The runner host must expose `/dev/kvm`. Whether GitHub-hosted runners in `e2e.yml` support KVM is the main open question before this test can be implemented.

## Notes

The existing `test-copilot-network-isolation.md` tests sandbox network policy but not the runtime layer. The open suggestion #6559 covers gvisor. This test specifically exercises the new `docker-sbx` path from github/gh-aw#45006.

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `awmgmcpg`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "awmgmcpg"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/29186981298) · 17.7 AIC · ⌖ 5.4 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test-copilot-docker-sbx-runtime.md, the existing test-copilot-network-isolation.md, and e2e.yml; first verify whether the runners expose /dev/kvm and can support the required Docker Hub secrets. Add the workflow only if that infrastructure is available, then confirm it creates the fixed-title issue and exercises the docker-sbx runtime end to end.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, shell
Domain
ci-cd, infrastructure, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.