openai / openai/openai-openapi

bug in YAML in property testing_criteria of type Eval

Open
#466 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug specification
Dominant language
No language data
Stars
2.5k
Forks
527
Avg merge
1h 46m
Merged PRs (30d)
2

Description

In

    Eval:
      type: object
      title: Eval
      description: |
        An Eval object with a data source config and testing criteria.
        An Eval represents a task to be done for your LLM integration.
        Like:
         - Improve the quality of my chatbot
         - See how well my chatbot handles customer support
         - Check if o3-mini is better at my usecase than gpt-4o
      properties:
        object:
          type: string
          enum:
            - eval
          default: eval
          description: The object type.
          x-stainless-const: true
        id:
          type: string
          description: Unique identifier for the evaluation.
        name:
          type: string
          description: The name of the evaluation.
          example: Chatbot effectiveness Evaluation
        data_source_config:
          type: object
          description: Configuration of data sources used in runs of the evaluation.
          oneOf:
            - $ref: "#/components/schemas/EvalCustomDataSourceConfig"
            - $ref: "#/components/schemas/EvalStoredCompletionsDataSourceConfig"
        testing_criteria:
          default: eval
          description: A list of testing criteria.
          type: array
          items:
            oneOf:
              - $ref: "#/components/schemas/EvalLabelModelGrader"
              - $ref: "#/components/schemas/EvalStringCheckGrader"
              - $ref: "#/components/schemas/EvalTextSimilarityGrader"
              - $ref: "#/components/schemas/EvalPythonGrader"
              - $ref: "#/components/schemas/EvalScoreModelGrader"

default: eval seems like a bug:

        testing_criteria:
          default: eval
          description: A list of testing criteria.
          type: array
          items:
            oneOf:
              - $ref: "#/components/schemas/EvalLabelModelGrader"
              - $ref: "#/components/schemas/EvalStringCheckGrader"
              - $ref: "#/components/schemas/EvalTextSimilarityGrader"
              - $ref: "#/components/schemas/EvalPythonGrader"
              - $ref: "#/components/schemas/EvalScoreModelGrader"

The property is of array type, and the array could contain one of the nominated types. So what is default value "eval"?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the Eval schema in the OpenAPI YAML and inspect the testing_criteria property against its array type and listed oneOf items. Validate the specification after the correction; done means testing_criteria no longer declares an invalid default and the document remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.