antirez / antirez/ds4

Add support for structured outputs

Offen
#210 12 Kommentare 3 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

As in https://docs.vllm.ai/en/latest/features/structured_outputs/#online-serving-openai-api and https://developers.openai.com/api/docs/guides/structured-outputs

I tried it with `ds4` using the latest code from `main` but it doesn't seem to be working fine (both DGX Spark and m3 max) -- here's the test:

```python
#!/usr/bin/env python3
from openai import OpenAI
from pydantic import BaseModel, ConfigDict

BASE_URL = "http://127.0.0.1:8000/v1"
API_KEY = "not-needed"
MODEL = "default"

class CalendarEvent(BaseModel):
model_config = ConfigDict(extra="forbid")

name: str
date: str
participants: list[str]

def main() -> None:
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)

print(f"[so-test] base_url={BASE_URL}")
print(f"[so-test] model={MODEL}")
print("[so-test] calling client.responses.parse(..., text_format=CalendarEvent)")

response = client.responses.parse(
model=MODEL,
input=[
{"role": "system", "content": "Extract the event information."},
{
"role": "user",
"content": "Alice and Bob are going to a science fair on Friday.",
},
],
text_format=CalendarEvent,
)

event = response.output_parsed
print("[so-test] parsed object:")
print(event.model_dump_json(indent=2))
print("[so-test] raw response id:", getattr(response, "id", None))

if __name__ == "__main__":
main()
```

Output:

```bash
$ python3 so-test-cli.py
[so-test] base_url=http://127.0.0.1:8000/v1
[so-test] model=default
[so-test] calling client.responses.parse(..., text_format=CalendarEvent)
Traceback (most recent call last):
[..]
pydantic_core._pydantic_core.ValidationError: 6 validation errors for CalendarEvent
people
Extra inputs are not permitted [type=extra_forbidden, input_value=['Alice', 'Bob'], input_type=list]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
event
Extra inputs are not permitted [type=extra_forbidden, input_value='science fair', input_type=str]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
day
Extra inputs are not permitted [type=extra_forbidden, input_value='Friday', input_type=str]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
name
Field required [type=missing, input_value={'people': ['Alice', 'Bob... fair', 'day': 'Friday'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
date
Field required [type=missing, input_value={'people': ['Alice', 'Bob... fair', 'day': 'Friday'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
participants
Field required [type=missing, input_value={'people': ['Alice', 'Bob... fair', 'day': 'Friday'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
```

A bit swamped but happy to look into this at some point

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.