microsoft / microsoft/promptpex
Results Overview shows NaN when eval metric score is not a number
Open
Nobody has claimed this yet.
bug
- Dominant language
- TeX
- Stars
- 164
- Forks
- 21
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 1
Description
When the eval metric returns something other than a number for the score, the summarizer for the eval results shows NaN:
<html>
<body>
<!--StartFragment-->
model | tests | tests positive | accuracy with eval
-- | -- | -- | --
qwen2.5:3b | 20 | 20 | 100
llama3.2:1b | 20 | 20 | NaN
<!--EndFragment-->
</body>
</html>
Eval metric:
<html>
<body>
<!--StartFragment-->
llama3.2:1b | Her voice resonated throughout the hall. Word: Her. | Incorporating the data .... | ❓ | The output must not provide explanations, descriptions, or reasoning and must consist solely of the applicable part-of-speech tag, "Unknown," or "CantAnswer."
-- | -- | -- | -- | --
<!--EndFragment-->
</body>
</html>Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.