Automate updating the calibration process
Open
Nobody has claimed this yet.
housekeeping
needs-triage
- Dominant language
- Python
- Stars
- 9.3k
- Forks
- 1.3k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 14
Description
Summary
Calibration involves a lot of manual work; update what can be updated
Basic example
Currently the process is:
- select a set of probes
- build a garak config based on that
- select a set of models
- run that config on all those models
- run the
calibrationscript on the results - add the output into
data/calibration - update
data/bag.md - update the
data/calibration.jsonpointer
Motivation
- To save time
- To make timely calibration updates more likely
- To improve consistency of our calibration
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing calibration script and the current flow from probe and model selection through config creation and results. Inspect data/calibration, data/bag.md, and data/calibration.json to understand the update steps. Done means the calibration process is repeatable with less manual work and keeps these outputs consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100