googleapis / googleapis/python-genai
Native to_openeval()/from_openeval() dataset conversion helpers (parity with an in-review openai-python PR)
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 40
Description
Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native `to_openeval()`/`from_openeval()` dataset conversion helpers directly into `openai-python`, in `src/openai/types/evals/openeval.py`:
https://github.com/openai/openai-python/pull/3619 (fixes https://github.com/openai/openai-python/issues/3549)
To be accurate about where it stands: it is **open, not merged** — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.
I'd like to propose something similar for `python-genai`, grounded in your actual types. From `google/genai/tests/transformers/test_t_content.py`, a turn is `types.Content(role=..., parts=[types.Part(text=...)])`, with helper constructors `types.UserContent` / `types.ModelContent`, and `types.Part` also carrying `function_call`, `inline_data`, etc. A rough sketch:
```python
# e.g. a new google.genai.openeval module
from google.genai import types
def to_openeval(contents: list[types.Content], *, id: str) -> dict:
"""Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
...
def from_openeval(item: dict) -> list[types.Content]:
"""Convert an EvalPort test_case back into Content/Part turns, e.g.
types.UserContent(parts=[types.Part(text=...)]) for a user turn,
types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
for a tool-call turn."""
...
```
Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.
Contributor guide
Assessment
This issue has not been assessed yet.