microsoft / microsoft/vscode-azureresourcegroups
CoR evaluation system
Open
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 78
- Forks
- 55
- Avg merge
- 16h 49m
- Merged PRs (30d)
- 139
Description
Wish list:
- Use vally
- Super detailed graders that define what quality projects are
- Run in CI/CD
- Run on many different project types and prompt variations
- Include baseline tests that run without Rails helpers (skills, custom agents, etc.)
- Test with multiple models
- Full test suite can run within 2 hours
- Have some sort of smaller test suite that can run more regularly on PRs or commits to main
- Evaluations should be presented in a nice report where it's easy to understand what failed. Can dive deeper into a specific prompt/project and see what went wrong
- Able to run a specific test slice locally for debugging/rapid development
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by examining the repository's existing test and CI setup, then map the wishlist to concrete components and agree on scope; done requires an approved evaluation design covering execution, reporting, test slices, and runtime constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100