HumanSignal / HumanSignal/label-studio
Optional EvalPort interop for tasks, annotations, and predictions
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
Hi Label Studio team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM evaluation datasets and results (a `TestCase`/`Suite`/`ResultSet` schema with a real validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as a feature-request issue first per CONTRIBUTING.md before writing any code.
I installed `label-studio-sdk` (2.1.1) and read the real types rather than guessing:
```python
# label_studio_sdk/types/lse_task.py
class LseTask(UncheckedBaseModel):
id: typing.Optional[int] = None
data: typing.Dict[str, typing.Any] = pydantic.Field() # user-imported task data
ground_truth: typing.Optional[bool] = None
annotations: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None
...
# label_studio_sdk/types/prediction.py
class Prediction(UncheckedBaseModel):
id: typing.Optional[int] = None
task: int
result: typing.List[typing.Dict[str, typing.Any]] = pydantic.Field()
score: typing.Optional[float] = None
model_version: typing.Optional[str] = None
# label_studio_sdk/types/annotation.py
class Annotation(UncheckedBaseModel):
id: typing.Optional[int] = None
ground_truth: typing.Optional[bool] = None
...
```
A project's `Task.data` maps to an EvalPort `TestCase.input`, a `ground_truth=True` `Annotation` on that task maps to `TestCase.expected`, and a `Prediction` (with its `result` and numeric `score`) maps to an EvalPort grader result for that test case — so a labeled Label Studio project with model predictions round-trips into an EvalPort `Suite` + `ResultSet` instead of a one-off export script per downstream tool. This seems particularly useful for the growing set of LLM-output-review projects people build in Label Studio (comparing predictions against human-annotated ground truth) rather than the classic CV/NLP annotation case.
Two ways I could see this landing, and I don't have a strong preference — happy to do whichever you'd rather maintain:
1. **A standalone `label-studio-openeval-adapter` package** in the EvalPort repo (same shape as the existing adapters there — a `to_openeval()`/`from_openeval()` pair depending on `label-studio-sdk` as a normal package dependency). Zero footprint on this repo beyond maybe a link from your docs if you're open to it.
2. **A small optional module inside `label-studio-sdk`** if you'd rather it live there instead, gated so it doesn't affect anyone not using it.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Contributor guide
Research direction
Start with CONTRIBUTING.md and the SDK models in label_studio_sdk/types/lse_task.py, prediction.py, and annotation.py. Confirm with maintainers whether this belongs in a standalone adapter or an optional SDK module, then define the round-trip scope and run tests against EvalPort’s actual JSON Schema. Done means the selected integration handles the stated task, annotation, and prediction mappings with real schema validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100