microsoft/promptpex

View on GitHub

Test Generation for Prompts

Stars: 164
Forks: 21
Open beginner issues: 0
Indexed issues: 15
Avg merge: 1d 13h
Merged PRs (30d): 1

Dominant language: TeX
License: CC-BY-4.0
Last GitHub push: Aug 25, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

15 open issues indexed

Proposal: promptpex-openeval-adapter — export generated tests/results to EvalPort format Open

Difficulty 5/5 Over a week Newbie friendliness 42/100

microsoft/promptpex#279 · Opened Aug 31, 2026
docs: edit-page links target nonexistent dev branch Open

Difficulty 1/5 1-3 hours Newbie friendliness 88/100

microsoft/promptpex#278 · Opened Aug 27, 2026
implement basic python layer Open

microsoft/promptpex#177 · 1 reaction · 1 assignee · Opened Jun 20, 2025
Results Overview shows NaN when eval metric score is not a number Open

bug

microsoft/promptpex#174 · 1 comment · 1 reaction · 1 assignee · Opened Jun 20, 2025
generating JSON inputs confuses the CSV output Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

microsoft/promptpex#165 · Opened Jun 16, 2025
Rules model reporting is incorrect Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

microsoft/promptpex#161 · Opened Jun 11, 2025
testdata is lost with github models Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

microsoft/promptpex#158 · Opened Jun 10, 2025
add ground truth evaluation Open

microsoft/promptpex#157 · 1 assignee · Opened Jun 10, 2025
Metrics are currently saved in the context. Open

bug

microsoft/promptpex#152 · 1 assignee · Opened Jun 6, 2025
add documentation page on ground truth Open

microsoft/promptpex#149 · 1 reaction · 2 assignees · Opened Jun 6, 2025
Disk button icon saves as .txt Open

microsoft/promptpex#131 · 1 reaction · 1 assignee · Opened May 22, 2025
support output specification in prompty Open

enhancement

microsoft/promptpex#51 · 1 reaction · 2 assignees · Opened Feb 20, 2025
Handle chat prompts that assume a multi-step interaction between user and system Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/promptpex#25 · Opened Oct 25, 2024
Feature request: Use the LLM to tag rules that are checkable using code Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/promptpex#9 · Opened Aug 28, 2024
Feature request: separate input/output rules into rules related to format/syntax and rules related to content Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/promptpex#8 · Opened Aug 28, 2024

microsoft/promptpex

Get new issues in your inbox