aws / aws/fmeval

Community package: fmeval-openeval-adapter — export datasets/results to a portable interchange format

Open
#353 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Hi — not a bug report or feature request, just flagging a small community package in case it's useful to anyone here.

I maintain [EvalPort](https://github.com/adhabnr-ux/evalport)/OpenEval, a small open interchange format (JSON Schema + a Python SDK, `evalport-sdk` on PyPI) for portable LLM evaluation datasets and results. The idea: eval data produced by one framework can be read, diffed, or re-checked by another tool without re-running anything. It's adapter-based — each supported framework gets a thin, zero-footprint conversion package that lives in `evalport`'s own repo, with no changes needed to the framework itself.

I just built and tested `fmeval-openeval-adapter` against fmeval's real API — `fmeval.data_loaders.data_config.DataConfig` for the dataset side, and the real per-record JSON Lines fmeval writes to `EvalOutput.output_path` for the results side:

```python
from fmeval.eval_algorithms.factual_knowledge import FactualKnowledge, FactualKnowledgeConfig
from fmeval_openeval_adapter import eval_output_to_openeval

outputs = FactualKnowledge(FactualKnowledgeConfig()).evaluate(model=None, dataset_config=config, save=True)
result_set = eval_output_to_openeval(output_path=outputs[0].output_path, suite_id="my_suite", run_id="run-1")

from openeval.validate import validate_result_set
assert validate_result_set(result_set).valid
```

A few real fmeval conventions I found and handled while building it: `toxicity` and `word_error_rate` are "lower is better" (the reverse of every other fmeval score), and `log_probability_difference` is a signed, unbounded log-probability delta — that one and `word_error_rate` get clamped into EvalPort's required `[0, 1]` score range with the true raw value always preserved alongside. 19 tests run end-to-end against a real installed `fmeval` (including a real `FactualKnowledge` evaluation run), passing in two independent clean-venv installs.

Package: https://github.com/adhabnr-ux/evalport/tree/main/adapters/fmeval-openeval-adapter
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Not asking for any change to fmeval itself — just leaving this here in case it's useful to anyone evaluating with fmeval who wants portable, framework-agnostic output. Happy to answer questions, or adjust the field mapping if it doesn't match how the team would want it to work.

— Sahi, independent contributor (not affiliated with AWS)

Contributor guide

Open the contributing guide

Research direction

The issue points to the external adapter at adapters/fmeval-openeval-adapter and the specification in spec/SPEC.md; start by reviewing those links and the described fmeval APIs and output path. It does not request a change or define a completion condition, so no implementation target is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.