googleapis / googleapis/python-genai

Native to_openeval()/from_openeval() dataset conversion helpers (parity with an in-review openai-python PR)

Offen
#2,901 2 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @kkorpal Auf GitHub ansehen
priority: p2 type: feature request
Vorherrschende Sprache
Python
Sterne
4k
Forks
1k
Ø Merge
2 T. 12 Std.
Gemergte PRs (30 T.)
41

Beschreibung

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.

Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native `to_openeval()`/`from_openeval()` dataset conversion helpers directly into `openai-python`, in `src/openai/types/evals/openeval.py`:

https://github.com/openai/openai-python/pull/3619 (fixes https://github.com/openai/openai-python/issues/3549)

To be accurate about where it stands: it is **open, not merged** — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.

I'd like to propose something similar for `python-genai`, grounded in your actual types. From `google/genai/tests/transformers/test_t_content.py`, a turn is `types.Content(role=..., parts=[types.Part(text=...)])`, with helper constructors `types.UserContent` / `types.ModelContent`, and `types.Part` also carrying `function_call`, `inline_data`, etc. A rough sketch:

```python
# e.g. a new google.genai.openeval module

from google.genai import types

def to_openeval(contents: list[types.Content], *, id: str) -> dict:
"""Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
...

def from_openeval(item: dict) -> list[types.Content]:
"""Convert an EvalPort test_case back into Content/Part turns, e.g.
types.UserContent(parts=[types.Part(text=...)]) for a user turn,
types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
for a tool-call turn."""
...
```

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.