googleapis / googleapis/python-genai

Native to_openeval()/from_openeval() dataset conversion helpers (parity with an in-review openai-python PR)

Abierto
#2,901 2 comentarios 0 reacciones 1 asignado Reclamado por @kkorpal Ver en GitHub
priority: p2 type: feature request
Lenguaje dominante
Python
Estrellas
4k
Forks
1k
Merge medio
2 d 12 h
PR fusionados (30 d)
41

Descripción

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.

Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native `to_openeval()`/`from_openeval()` dataset conversion helpers directly into `openai-python`, in `src/openai/types/evals/openeval.py`:

https://github.com/openai/openai-python/pull/3619 (fixes https://github.com/openai/openai-python/issues/3549)

To be accurate about where it stands: it is **open, not merged** — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.

I'd like to propose something similar for `python-genai`, grounded in your actual types. From `google/genai/tests/transformers/test_t_content.py`, a turn is `types.Content(role=..., parts=[types.Part(text=...)])`, with helper constructors `types.UserContent` / `types.ModelContent`, and `types.Part` also carrying `function_call`, `inline_data`, etc. A rough sketch:

```python
# e.g. a new google.genai.openeval module

from google.genai import types

def to_openeval(contents: list[types.Content], *, id: str) -> dict:
"""Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
...

def from_openeval(item: dict) -> list[types.Content]:
"""Convert an EvalPort test_case back into Content/Part turns, e.g.
types.UserContent(parts=[types.Part(text=...)]) for a user turn,
types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
for a tool-call turn."""
...
```

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.