Capgemini / Capgemini/gov-prototype-by-prompt

Run LLM evals in GitHub Actions

Open
#129 0 comments 0 reactions 0 assignees View on GitHub
size M
Dominant language
TypeScript
Stars
20
Forks
4
PR merge metrics
No merged PRs in 30d

Description

### What feature do you want to see added?

Run the LLM evals in GitHub Actions, like the unit tests.

### Why do you want this feature? Does it solve a problem?

This would allow us to showcase the robustness of the tool.

### How should the feature be implemented?

Similar to how existing tests are run.

Consider how to ensure LLM usage can be monitored and restricted.

Consider how to show failures, given that the LLM evals are (by their nature) flaky.

### Additional context (if applicable)

Follows on from #111 and #127.

Contributor guide

Open the contributing guide

Research direction

Start by reading issues #111 and #127, then inspect the existing GitHub Actions setup used for unit tests. Determine how LLM evals should run there, how usage can be monitored and restricted, and how flaky failures should be reported. Done means the workflow runs the evals with those safeguards and communicates failures clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.