dotnet / dotnet/extensions

[AI Evaluation] Support evaluation of JSONL dataset files in the dotnet aieval command line tool

Open
#6,482 0 comments 0 reactions 1 assignee Claimed by @shyamnamboodiripad View on GitHub
area-ai-eval
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

The idea would be to allow evaluation of question-answer pairs without needing to author unit tests for them. In addition to question-answer pairs to be evaluated, the JSONL dataset could also include the list of evaluators to be executed + the configuration for setting up LLM / Azure AI Foundry evaluation service connection. It could then internally execute the configured evaluators against each question-answer pair and produce a report containing all the evaluation scores.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.