[AI Evaluation] Support evaluation of JSONL dataset files in the dotnet aieval command line tool
Open
area-ai-eval
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
The idea would be to allow evaluation of question-answer pairs without needing to author unit tests for them. In addition to question-answer pairs to be evaluated, the JSONL dataset could also include the list of evaluators to be executed + the configuration for setting up LLM / Azure AI Foundry evaluation service connection. It could then internally execute the configured evaluators against each question-answer pair and produce a report containing all the evaluation scores.
Contributor guide
Assessment
This issue has not been assessed yet.