Add safety and fairness benchmarks

Goal

Integrate the safety and fairness benchmarks listed in the benchmark matrix into oellm-eval.

Implementation plan

Benchmarks differ in their applicability to pretrained, instruction-tuned, and safety-tuned models. We will begin with benchmarks suitable for pretrained/base models, followed by those that require instruction-following or safety-tuned behavior.

This work also includes adding or extending support for inspect_evals and lighteval benchmarks where necessary.

Implementation progress

Shared infrastructure

• Add and verify inspect_evals integration in oellm-eval

lm-eval

• CrowS-Pairs
• ToxiGen
• BBQ

inspect_evals

• StereoSet
• WMDP
• BBQ
• BOLD
• XSTest
• StrongREJECT
• AgentHarm
• AgentDojo

lighteval

• RealToxicityPrompts

Custom or external integrations

These benchmarks are not currently recorded as implementations in the three suites above and may require custom adapters or their native evaluation code.

• HarmBench
• AdvBench
• JailbreakBench

Recommended benchmarks by model stage

Phase 1: Pretrained/base models

• CrowS-Pairs
• StereoSet
• RealToxicityPrompts
• ToxiGen
• WMDP
• BBQ, using likelihood-based scoring where appropriate

Refusal and jailbreak scores should not be interpreted as safety-alignment measurements for base models.

Phase 2: Instruction-tuned/SFT models

• BBQ
• BOLD
• RealToxicityPrompts
• ToxiGen
• XSTest
• HarmBench
• StrongREJECT
• AdvBench

Phase 3: Post-trained/safety-tuned models

• XSTest
• HarmBench
• JailbreakBench
• StrongREJECT
• AgentHarm
• AgentDojo

Definition of done

For each benchmark:

• The benchmark is runnable through the standard oellm-eval workflow
• The upstream suite and task name are documented
• A minimal configuration or example command is provided
• A smoke test verifies that the benchmark loads and produces results
• Results are written in the standard oellm-eval output format
• Applicable model stages and interpretation caveats are documented

A benchmark may appear in more than one upstream suite. Each suite integration should be tracked separately when the implementations or resulting metrics differ.

Start with the benchmark matrix and inspect the existing oellm-eval workflow, then determine how lm-eval, inspect_evals, lighteval, and custom integrations are represented. No repository files or tests are named, so first locate the integration entry points and smoke-test conventions. Done means each selected benchmark runs through the standard workflow, has documented suite/task details and configuration, produces standard output, and records model-stage caveats.