Author eval suite for agent `azure-template-generator`
- Dominant language
- JavaScript
- Stars
- 269
- Forks
- 45
- Avg merge
- 18h 41m
- Merged PRs (30d)
- 11
Description
## Agent
`azure-template-generator` — source: `.github/agents/azure-template-generator.agent.md`
## Scope
Author the eval suite at `.github/evals/agents/azure-template-generator/`:
- [ ] `eval.yaml` — suite config (executor, model, graders)
- [ ] At least 2 positive tasks under `tasks/positive-*.yaml`
- [ ] At least 1 negative task under `tasks/negative-*.yaml`
- [ ] Entry added to `.github/evals/manifest.yaml` at `tier: expanded`
## Notes
Positive tasks that generate ARM templates via `create` should ALSO require the agent to paste the rendered JSON inline in a fenced code block in its chat response. `output_contains` graders only check chat output, not files created via `create` — without an inline echo the grader can't audit the template.
## Procedure
1. `/agent-bench azure-template-generator` drafts the suite from the live `.agent.md`.
2. `waza run .github/evals/agents/azure-template-generator/eval.yaml -v` locally.
3. `/agent-improve azure-template-generator` to iterate on graders.
4. Open PR.
5. Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.
## Acceptance
- [ ] Suite runs cleanly in `mock` executor.
- [ ] At least one positive task passes in a real-model run.
- [ ] All negative tasks produce a refusal or out-of-scope acknowledgement.
- [ ] `manifest.yaml` entry added; PR description includes the real-model run summary.
## Conventions to follow
- Persona lock: refusal graders should accept the agent's own scope language.
- Prompt graders need `continue_session: true` in their grader config.
## Related
- Umbrella: #93
- Harness: #61
Contributor guide
Assessment
This issue has not been assessed yet.