microsoft / microsoft/vscode-azureresourcegroups
Port the Podman plan-gate stimulus to MSBench so the runtime path actually executes
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 78
- Forks
- 55
- Avg merge
- 16h 49m
- Merged PRs (30d)
- 139
Description
Follow-up from the review of #1764. Not a defect in that PR — the validator side is sound and tested — but the Podman path currently has grader capability and zero execution, and that gap is invisible from a green check run.
What's missing
#1764 adds debug-plan-podman-runtime to evals/local-dev/eval.yaml. That spec has no executor: key, so it is lint-validated only — the agent never runs against it. Both of its siblings have MSBench ports; this one does not:
| stimulus | in local-dev/eval.yaml |
MSBench port |
|---|---|---|
debug-plan-approval-gate |
yes | msbench/config/stimuli/debug-plan-approval-gate.yaml |
debug-generate-artifacts |
yes | msbench/config/stimuli/debug-generate-artifacts.yaml |
debug-plan-podman-runtime |
yes | none |
Why it's worth doing
The --assert-runtime / --assert-docker-compat flags work. I verified 13 cases against purpose-built fixtures, including the backward-compat pair (an old single-column plan passes with no flags and fails when asked to prove a runtime it has no data for) and exit-3 on miswired flags.
But all of that is a statement about the grader, not about what the agent writes when a user says "I use Podman, not Docker Desktop". Nothing currently exercises:
- whether the agent records
podmanin the Orchestrator table when asked - whether the generated
Start Emulatorstask usespodman compose - whether the lifecycle-wide "never switch the runtime silently" rule in
preflight.mdholds under a real failure
That last one matters most. It was added in 4ec0a7b because the agent was observed silently switching a user-requested Podman plan to Docker. The rule is written down; nothing regression-tests it.
The work
Port debug-plan-podman-runtime the way its siblings were ported — a msbench/config/stimuli/redteam-style two-turn stimulus with # phase: local and # seed: approved-fullstack, carrying:
- the liveness sentinel (every stimulus needs one)
validate-debug-plan.ts --assert-runtime=podman- the approval-gate check, as in
debug-plan-approval-gate.yaml open_local_plan_viewtool-call assertion
The stimulus stops at the approval gate, so Podman does not need to be installed in the eval environment — it only grades the plan the agent wrote. That's what makes this cheap to run.
Verify with npm run gates (assertion-comment identity) and one live run before trusting the result.
Secondary: the compose-command conformance check is evadable
Recording this here so it isn't lost — it's small and in the same area.
validateComposeCommandConformance in evals/src/artifacts/debugArtifacts.ts reads task.command and matches \b(docker|podman)[ -]compose\b. VS Code also accepts the split form, and that form slips through. Measured on a Podman plan with a docker compose task written both ways:
command: "docker compose up -d" exit=1 CAUGHT
command: "docker", args: ["compose","up","-d"] exit=0 not caught
generate.md §171 directs the whole command into command, so the documented shape is the one that gets caught and this is unlikely in practice. But the check's coverage depends on the agent choosing one of two valid encodings, and nothing enforces that choice. Reading args when command is a bare engine name would close it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the sibling stimuli in msbench/config/stimuli/debug-plan-approval-gate.yaml and debug-generate-artifacts.yaml, then inspect evals/local-dev/eval.yaml and evals/src/artifacts/debugArtifacts.ts. Port the Podman stimulus with its stated assertions, run npm run gates, and perform one live run; done means the plan is graded for Podman runtime behavior and the conformance check covers the documented command form.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript, yaml
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100