Publish reusable skills for community template validation
- Dominant language
- TypeScript
- Stars
- 269
- Forks
- 176
- Avg merge
- 11h 1m
- Merged PRs (30d)
- 37
Description
**Is your feature request related to a problem? Please describe.**
Community template submissions currently require validation by the manual test team. However, the manual test team may not always have the capacity or scope to test every community contribution. When manual testing is unavailable, having skills that contributors could run is helpful.
@v-xuto Could the manual test team help identify the repeatable parts of the current community-template validation process that could be shared as reusable agent skills?
**Describe the solution you'd like**
Document the current validation checklist and package deterministic, safe-to-automate checks as one or more reusable skills. The skills should allow a contributor, reviewer, or agent to run applicable tests locally before requesting manual validation.
The goal is not to replace manual testing entirely. Automated skills should cover repeatable checks, while scenarios requiring human judgment, specialized environments, or credentials that cannot safely be used locally should remain with the manual test team.
Proposed work:
1. Document the current validation checklist for community templates.
2. Identify checks that are deterministic and safe to run locally or in an approved agent environment.
3. Package those checks as one or more reusable skills.
4. Clearly document prerequisites, supported template types, required credentials, cleanup expectations, and expected outputs.
5. Produce a concise report that can be attached to the template PR, distinguishing passed, failed, and skipped checks.
6. Document which scenarios must still be escalated to the manual test team.
7. Validate and evaluate each skill with Valley, or other skill-evaluation tool, before publishing it. Evaluation should cover instruction quality, reliability, repeatability, safety, and representative success and failure scenarios.
Example checks could include:
- Initialize the template with `azd`.
- Validate generated project and infrastructure files.
- Run documented build and test commands.
- Exercise `azd provision` and `azd deploy` when credentials and an isolated subscription are available.
- Verify documented outputs and endpoints.
- Run cleanup and confirm that created resources are removed.
- Report skipped checks and why they could not be run.
Acceptance criteria:
- At least one reusable skill covers the common community-template validation path.
- The skill can run locally or in an approved agent environment.
- Results distinguish passed, failed, and skipped checks, with actionable diagnostics for failures.
- Secrets and customer-identifying information are excluded from logs and reports.
- Documentation explains when manual test-team involvement is still required.
- Each published skill passes validation and evaluation using Valley or another approved skill-evaluation tool.
- Evaluation results cover representative success, failure, and skipped-check scenarios.
**Additional context**
This would provide a fallback when the manual test team cannot take a request: an agent could run the supported checks locally and attach the resulting report to the PR, while clearly identifying anything that still requires manual validation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by documenting the current community-template validation checklist and mapping its repeatable checks to the proposed azd init, build, test, provision, deploy, output-verification, and cleanup steps. Define prerequisites, credential and safety boundaries, passed/failed/skipped reporting, manual-test escalation, and evaluate each published skill with Valley or another approved evaluation tool.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cli, cloud, documentation, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 40/100