Interop idea: a VLMEvalKit-openeval-adapter for portable eval results

EvalPort (https://github.com/adhabnr-ux/evalport) is a small open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM/LMM evaluation datasets and results — a JSON schema plus openeval.validate.validate_suite() / validate_result_set(), so results from one eval toolkit can be read, diffed, or re-checked by another without re-running anything. It's adapter-based: each framework gets a thin, zero-footprint adapters/<name>-openeval-adapter/ package converting to/from the interchange shape — 34 exist today. (Note: this is specifically about VLMEvalKit, not opencompass — those are separate repos/packages and opencompass already has its own adapter.)

Why VLMEvalKit fits, based on the real shape I read in vlmeval/dataset/image_base.py and vlmeval/dataset/image_yorn.py: every dataset subclasses ImageBaseDataset and implements the abstract evaluate(self, eval_file, **judge_kwargs). eval_file is a per-instance prediction table (xlsx/tsv, index/question/prediction/answer and friends) written after inference; evaluate() scores it per instance (e.g. ImageYORNDataset.evaluate sets data['score'] = (data['answer'] == data['extracted'])) and returns an aggregate dict/DataFrame (e.g. default_rating(storage), dumped to {eval_file}_score.csv). So every dataset already produces both a per-instance result table and an aggregate score summary — just in VLMEvalKit's own file conventions rather than a shared interchange format.

Sketch:

# after: python run.py --data MME --model Qwen2-VL-7B
# produces Qwen2-VL-7B_MME.xlsx (per-instance predictions/scores)
# and      Qwen2-VL-7B_MME_score.csv (aggregate, via evaluate())

# proposed: vlmevalkit_openeval_adapter
from vlmevalkit_openeval_adapter import results_to_openeval
result_set = results_to_openeval(
    prediction_file="Qwen2-VL-7B_MME.xlsx",
    score_file="Qwen2-VL-7B_MME_score.csv",
)

from openeval.validate import validate_result_set
validate_result_set(result_set)

Honest caveat: the per-instance table's columns vary by dataset TYPE (MCQ vs Y/N vs VQA vs generative), and image/image_path hold base64-encoded images or local paths rather than portable references. So the adapter I'd build maps the columns common across dataset classes (index, question, prediction, answer/extracted, score) and represents image_path as a reference rather than re-encoding pixel data — full multimodal fidelity is a real stretch for EvalPort's current text-first model, and I'd rather say that upfront than oversell it.

If useful, I'll build adapters/vlmevalkit-openeval-adapter/ (zero footprint on VLMEvalKit, pyproject convention matching the other 34 adapters — self-referencing pinned extra + test extra, per evalport Discussion #13) and submit it as a PR to evalport, referencing this issue.

— Sahi, independent contributor (not affiliated with open-compass)

Start by reading vlmeval/dataset/image_base.py and vlmeval/dataset/image_yorn.py to compare per-instance fields and evaluate() outputs. Then inspect the adapters/-openeval-adapter/ convention and EvalPort's validation guidance in Discussion #13. Done means a zero-footprint adapter converts the named VLMEvalKit result files and its output passes validate_result_set(), with tests covering the shared fields and image references.