Optional EvalPort interop for Datalab label-quality results
- Dominant language
- Python
- Stars
- 11.7k
- Forks
- 920
- PR merge metrics
- No merged PRs in 30d
Description
Hi cleanlab team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a `TestCase`/`Suite`/`ResultSet` schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.
I installed `cleanlab` (2.9.0) and ran the real `Datalab` API rather than guessing:
```python
lab = Datalab(data=data, label_name="labels", task="classification")
lab.find_issues(pred_probs=pred_probs)
lab.get_issues()
```
`get_issues()` returns one row per example with columns like `is_label_issue` (bool), `label_score` (float, 0–1), plus the same `is_*_issue` / `*_score` pair for every other issue type that ran (`outlier`, `non_iid`, `class_imbalance`, etc.). `get_issue_summary()` aggregates those into `issue_type` / `score` / `num_issues` rows.
That's a very direct match for EvalPort's other half: each example in the audited dataset is a `TestCase`, and the `is_label_issue` + `label_score` (or any other issue-type's bool+score pair) that `Datalab` computes per example is exactly the shape of an EvalPort `Grader` result — a per-`TestCase` pass/fail plus a numeric score — so `Datalab.get_issues()` output could round-trip into an EvalPort `ResultSet` with very little translation.
Two ways I could see this landing, and I don't have a strong preference:
1. **A standalone `cleanlab-openeval-adapter` package** in the EvalPort repo, depending on `cleanlab` as a normal dependency. Zero footprint on this repo.
2. **A small optional module inside this repo** if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Contributor guide
Research direction
Start with CONTRIBUTING.md and the Datalab API calls shown in the issue: find_issues(), get_issues(), and get_issue_summary(). Compare their real output with EvalPort’s actual JSON Schema, then resolve whether the adapter belongs in a standalone package or this repository. Done means the placement is agreed and real-schema tests validate the ResultSet round trip.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100