huggingface / huggingface/lighteval
Custom task: RAIL Score responsible AI evaluation (8-dimension scoring)
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Problem
Standard LLM benchmarks measure accuracy, fluency, and task completion but do not capture responsible AI dimensions like fairness, safety, privacy, or accountability. Teams deploying models in production need structured evaluation across these dimensions.
## What RAIL Score provides
[RAIL Score](https://responsibleailabs.ai/) is a responsible AI evaluation API that scores model outputs across **8 dimensions**, each on a 0-10 scale:
| Dimension | What it measures |
|-----------|-----------------|
| Fairness | Equitable treatment, absence of bias |
| Safety | Prevention of harmful content |
| Reliability | Factual accuracy, consistency |
| Transparency | Clear reasoning, disclosed limitations |
| Privacy | PII protection, data minimization |
| Accountability | Traceable decisions, auditable reasoning |
| Inclusivity | Accessible, culturally aware language |
| User Impact | Value delivered to the end user |
Python SDK on PyPI: `pip install rail-score-sdk`
## Integration with LightEval
Uses `MetricGrouping` with a `SampleLevelComputation` subclass. All 8 dimensions + overall score appear as **separate named metrics** in LightEval results.
```bash
pip install rail-score-sdk
export RAIL_API_KEY="rail_..."
lighteval accelerate \
"model_name=HuggingFaceH4/zephyr-7b-beta" \
"rail_score:default|0" \
--custom-tasks custom_rail_score_task.py
```
A complete custom task file is available in the linked PR.
## Resources
- [RAIL Score SDK on PyPI](https://pypi.org/project/rail-score-sdk/)
- [SDK Documentation](https://docs.responsibleailabs.ai)
- [Free API key signup](https://responsibleailabs.ai)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.