cleanlab / cleanlab/cleanlab

Optional EvalPort interop for Datalab label-quality results

Open
#1,334 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
920
PR merge metrics
No merged PRs in 30d

Description

Hi cleanlab team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a `TestCase`/`Suite`/`ResultSet` schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.

I installed `cleanlab` (2.9.0) and ran the real `Datalab` API rather than guessing:

```python
lab = Datalab(data=data, label_name="labels", task="classification")
lab.find_issues(pred_probs=pred_probs)
lab.get_issues()
```

`get_issues()` returns one row per example with columns like `is_label_issue` (bool), `label_score` (float, 0–1), plus the same `is_*_issue` / `*_score` pair for every other issue type that ran (`outlier`, `non_iid`, `class_imbalance`, etc.). `get_issue_summary()` aggregates those into `issue_type` / `score` / `num_issues` rows.

That's a very direct match for EvalPort's other half: each example in the audited dataset is a `TestCase`, and the `is_label_issue` + `label_score` (or any other issue-type's bool+score pair) that `Datalab` computes per example is exactly the shape of an EvalPort `Grader` result — a per-`TestCase` pass/fail plus a numeric score — so `Datalab.get_issues()` output could round-trip into an EvalPort `ResultSet` with very little translation.

Two ways I could see this landing, and I don't have a strong preference:

1. **A standalone `cleanlab-openeval-adapter` package** in the EvalPort repo, depending on `cleanlab` as a normal dependency. Zero footprint on this repo.
2. **A small optional module inside this repo** if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.

Contributor guide

Open the contributing guide

Research direction

Start with CONTRIBUTING.md and the Datalab API calls shown in the issue: find_issues(), get_issues(), and get_issue_summary(). Compare their real output with EvalPort’s actual JSON Schema, then resolve whether the adapter belongs in a standalone package or this repository. Done means the placement is agreed and real-schema tests validate the ResultSet round trip.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.