Community package: fmeval-openeval-adapter — export datasets/results to a portable interchange format
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Hi — not a bug report or feature request, just flagging a small community package in case it's useful to anyone here.
I maintain [EvalPort](https://github.com/adhabnr-ux/evalport)/OpenEval, a small open interchange format (JSON Schema + a Python SDK, `evalport-sdk` on PyPI) for portable LLM evaluation datasets and results. The idea: eval data produced by one framework can be read, diffed, or re-checked by another tool without re-running anything. It's adapter-based — each supported framework gets a thin, zero-footprint conversion package that lives in `evalport`'s own repo, with no changes needed to the framework itself.
I just built and tested `fmeval-openeval-adapter` against fmeval's real API — `fmeval.data_loaders.data_config.DataConfig` for the dataset side, and the real per-record JSON Lines fmeval writes to `EvalOutput.output_path` for the results side:
```python
from fmeval.eval_algorithms.factual_knowledge import FactualKnowledge, FactualKnowledgeConfig
from fmeval_openeval_adapter import eval_output_to_openeval
outputs = FactualKnowledge(FactualKnowledgeConfig()).evaluate(model=None, dataset_config=config, save=True)
result_set = eval_output_to_openeval(output_path=outputs[0].output_path, suite_id="my_suite", run_id="run-1")
from openeval.validate import validate_result_set
assert validate_result_set(result_set).valid
```
A few real fmeval conventions I found and handled while building it: `toxicity` and `word_error_rate` are "lower is better" (the reverse of every other fmeval score), and `log_probability_difference` is a signed, unbounded log-probability delta — that one and `word_error_rate` get clamped into EvalPort's required `[0, 1]` score range with the true raw value always preserved alongside. 19 tests run end-to-end against a real installed `fmeval` (including a real `FactualKnowledge` evaluation run), passing in two independent clean-venv installs.
Package: https://github.com/adhabnr-ux/evalport/tree/main/adapters/fmeval-openeval-adapter
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Not asking for any change to fmeval itself — just leaving this here in case it's useful to anyone evaluating with fmeval who wants portable, framework-agnostic output. Happy to answer questions, or adjust the field mapping if it doesn't match how the team would want it to work.
— Sahi, independent contributor (not affiliated with AWS)
Contributor guide
Research direction
The issue points to the external adapter at adapters/fmeval-openeval-adapter and the specification in spec/SPEC.md; start by reviewing those links and the described fmeval APIs and output path. It does not request a change or define a completion condition, so no implementation target is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100