microsoft / microsoft/vscode-azureresourcegroups

Port the Podman plan-gate stimulus to MSBench so the runtime path actually executes

Open
#1,777 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

copilot-on-rails
Dominant language
TypeScript
Stars
78
Forks
55
Avg merge
16h 49m
Merged PRs (30d)
139

Description

Follow-up from the review of #1764. Not a defect in that PR — the validator side is sound and tested — but the Podman path currently has grader capability and zero execution, and that gap is invisible from a green check run.

What's missing

#1764 adds debug-plan-podman-runtime to evals/local-dev/eval.yaml. That spec has no executor: key, so it is lint-validated only — the agent never runs against it. Both of its siblings have MSBench ports; this one does not:

stimulus in local-dev/eval.yaml MSBench port
debug-plan-approval-gate yes msbench/config/stimuli/debug-plan-approval-gate.yaml
debug-generate-artifacts yes msbench/config/stimuli/debug-generate-artifacts.yaml
debug-plan-podman-runtime yes none

Why it's worth doing

The --assert-runtime / --assert-docker-compat flags work. I verified 13 cases against purpose-built fixtures, including the backward-compat pair (an old single-column plan passes with no flags and fails when asked to prove a runtime it has no data for) and exit-3 on miswired flags.

But all of that is a statement about the grader, not about what the agent writes when a user says "I use Podman, not Docker Desktop". Nothing currently exercises:

  • whether the agent records podman in the Orchestrator table when asked
  • whether the generated Start Emulators task uses podman compose
  • whether the lifecycle-wide "never switch the runtime silently" rule in preflight.md holds under a real failure

That last one matters most. It was added in 4ec0a7b because the agent was observed silently switching a user-requested Podman plan to Docker. The rule is written down; nothing regression-tests it.

The work

Port debug-plan-podman-runtime the way its siblings were ported — a msbench/config/stimuli/redteam-style two-turn stimulus with # phase: local and # seed: approved-fullstack, carrying:

  • the liveness sentinel (every stimulus needs one)
  • validate-debug-plan.ts --assert-runtime=podman
  • the approval-gate check, as in debug-plan-approval-gate.yaml
  • open_local_plan_view tool-call assertion

The stimulus stops at the approval gate, so Podman does not need to be installed in the eval environment — it only grades the plan the agent wrote. That's what makes this cheap to run.

Verify with npm run gates (assertion-comment identity) and one live run before trusting the result.

Secondary: the compose-command conformance check is evadable

Recording this here so it isn't lost — it's small and in the same area.

validateComposeCommandConformance in evals/src/artifacts/debugArtifacts.ts reads task.command and matches \b(docker|podman)[ -]compose\b. VS Code also accepts the split form, and that form slips through. Measured on a Podman plan with a docker compose task written both ways:

command: "docker compose up -d"                    exit=1  CAUGHT
command: "docker", args: ["compose","up","-d"]     exit=0  not caught

generate.md §171 directs the whole command into command, so the documented shape is the one that gets caught and this is unlikely in practice. But the check's coverage depends on the agent choosing one of two valid encodings, and nothing enforces that choice. Reading args when command is a bare engine name would close it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sibling stimuli in msbench/config/stimuli/debug-plan-approval-gate.yaml and debug-generate-artifacts.yaml, then inspect evals/local-dev/eval.yaml and evals/src/artifacts/debugArtifacts.ts. Port the Podman stimulus with its stated assertions, run npm run gates, and perform one live run; done means the plan is graded for Podman runtime behavior and the conformance check covers the documented command form.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript, yaml
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.