googleapis / googleapis/python-genai

Native to_openeval()/from_openeval() dataset conversion helpers (parity with an in-review openai-python PR)

Open
#2,901 2 comments 0 reactions 1 assignee Claimed by @kkorpal View on GitHub
priority: p2 type: feature request
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.

Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native `to_openeval()`/`from_openeval()` dataset conversion helpers directly into `openai-python`, in `src/openai/types/evals/openeval.py`:

https://github.com/openai/openai-python/pull/3619 (fixes https://github.com/openai/openai-python/issues/3549)

To be accurate about where it stands: it is **open, not merged** — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.

I'd like to propose something similar for `python-genai`, grounded in your actual types. From `google/genai/tests/transformers/test_t_content.py`, a turn is `types.Content(role=..., parts=[types.Part(text=...)])`, with helper constructors `types.UserContent` / `types.ModelContent`, and `types.Part` also carrying `function_call`, `inline_data`, etc. A rough sketch:

```python
# e.g. a new google.genai.openeval module

from google.genai import types

def to_openeval(contents: list[types.Content], *, id: str) -> dict:
"""Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
...

def from_openeval(item: dict) -> list[types.Content]:
"""Convert an EvalPort test_case back into Content/Part turns, e.g.
types.UserContent(parts=[types.Part(text=...)]) for a user turn,
types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
for a tool-call turn."""
...
```

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.