kubernetes-sigs / kubernetes-sigs/devops-bench

Determine what type of eval framework we should use

Open
#3 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

There are many skill eval frameworks in the community such as https://github.com/promptfoo/promptfoo. I wonder if we are going to pick one framework or make the evel in this repo work for mulitple framewok?

Contributor guide

Open the contributing guide

Research direction

Review how evaluation currently works in this repository, then compare its needs with the promptfoo framework linked in the issue. Determine whether to adopt one framework or support multiple frameworks; done means the framework direction and its scope are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.