openai / openai/openai-python

Invalid Schema Error when Using examples Field in Pydantic Model for OpenAI Python Client

Open
#2,274 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

api bug sdk
Dominant language
Python
Stars
31.6k
Forks
5.7k
Avg merge
1d 6h
Merged PRs (30d)
96

Description

Confirm this is an issue with the Python library and not an underlying OpenAI API
  • This is an issue with the Python library
Describe the bug

When you add the examples field in the Field definition of a Pydantic model (as shown in the code below) while using the OpenAI Python client, it raises a BadRequestError with error code 400. The error message indicates that the examples field is not permitted in the schema for the response_format parameter.

This is the error message:

raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'MathReasoning': In context=('properties', 'final_answer'), 'examples' is not permitted.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}
Question:

Is this behavior due to a limitation in the OpenAI Python client or API, or is it something that could potentially be patched in the future?

To Reproduce
from openai import Client
from pydantic import BaseModel, Field
from dotenv import load_dotenv

load_dotenv()

client = Client()

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    steps: list[Step]
    final_answer: str = Field(
        title="Final Answer",
        description="The final answer to the math problem",
        examples=["x = -3", "x = 2"]
    )

completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step."},
        {"role": "user", "content": "how can I solve 8x + 7 = -23"}
    ],
    response_format=MathReasoning,
)

math_reasoning = completion.choices[0].message

# If the model refuses to respond, you will get a refusal message
if (math_reasoning.refusal):
    print(math_reasoning.refusal)
else:
    print(math_reasoning.parsed)
Code snippets

OS

Windows

Python version

Python v3.11.9

Library version

openai v1.68.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at client.beta.chat.completions.parse and inspect how the Pydantic response_format schema is sent. Reproduce the issue with the supplied MathReasoning model and determine whether the client or API rejects the examples field. Done means a documented decision on whether a client change is warranted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.