microsoft / microsoft/promptpex
Feature request: Use the LLM to tag rules that are checkable using code
Nobody has claimed this yet.
- Dominant language
- TeX
- Stars
- 164
- Forks
- 21
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 1
Description
We can leverage the LLM to determine what rules are checkable with code and then generate code to apply them to the test cases. Since there are probably a limited number of possible code checkers, we could have a set of pre-defined checkers that the LLM would just have to choose between (possibly).
We talked about this feature but didn't have time to add it.
Checkers can be generated both as input filters (rejecting inputs that don't meet the requirements) and as output filters.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Start by mapping the existing rule and test-case evaluation flow, then determine how predefined checkers could be selected or generated for both input and output filtering; done means those checkers are applied to test cases as described.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100