huggingface / huggingface/evaluate

New community metric: RAIL Score — responsible AI evaluation across 8 dimensions

Open
#742 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

## What

[RAIL Score](https://huggingface.co/spaces/responsible-ai-labs/rail_score) is a responsible AI evaluation metric now available as a community metric on the Hub. It scores LLM outputs across **8 dimensions** on a 0–10 scale:

| Dimension | What It Measures |
|-----------|-----------------|
| Fairness | Equitable treatment across demographic groups |
| Safety | Prevention of harmful or unsafe content |
| Reliability | Factual accuracy, internal consistency |
| Transparency | Clear communication of limitations and reasoning |
| Privacy | Protection of personal and sensitive data |
| Accountability | Traceability and auditability of decisions |
| Inclusivity | Accessible, inclusive language |
| User Impact | Positive value delivered to the user |

## Usage

```python
import evaluate

rail_score = evaluate.load("responsible-ai-labs/rail_score")

results = rail_score.compute(
predictions=["The capital of France is Paris."],
references=["What is the capital of France?"],
)

print(results["overall_score"]) # 8.2
print(results["overall_confidence"]) # 0.85
print(results["safety"]) # 9.5
print(results["fairness"]) # 8.0
```

## Features

- All 8 RAIL dimensions with per-dimension and per-example scores
- **Confidence scores** for every dimension and overall
- **Custom dimension weights** (e.g., prioritize safety over inclusivity)
- `basic` and `deep` evaluation modes (deep includes grounded explanations)
- Domain-specific scoring: `general`, `healthcare`, `finance`, `legal`, `education`, `code`
- Issue detection with dimension-level flagging

## Deep mode example

```python
results = rail_score.compute(
predictions=responses,
references=prompts,
mode="deep",
include_explanations=True,
include_issues=True,
weights={"safety": 25, "reliability": 20, "fairness": 15,
"transparency": 10, "privacy": 10, "accountability": 5,
"inclusivity": 10, "user_impact": 5},
)

# Per-dimension explanations
for dim, explanations in results["explanations"].items():
print(f"{dim}: {explanations[0]}")
```

## Links

- **Hub Space**: [responsible-ai-labs/rail_score](https://huggingface.co/spaces/responsible-ai-labs/rail_score)
- **SDK on PyPI**: [rail-score-sdk](https://pypi.org/project/rail-score-sdk/)
- **Documentation**: [docs.responsibleailabs.ai](https://docs.responsibleailabs.ai)

Requires a RAIL Score API key (free tier available at [responsibleailabs.ai](https://responsibleailabs.ai)).

Would be great to get this listed alongside other community evaluation metrics. Happy to make any changes needed for better integration with the evaluate ecosystem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.