openai / openai/openai-openapi

GraderMulti.graders models one grader instead of a named dictionary

Open
#580 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug documentation specification
Dominant language
No language data
Stars
2.5k
Forks
527
Avg merge
1h 46m
Merged PRs (30d)
2

Description

Source report

Mirrors openai-java#647, which reports a generated MultiGrader request failing with 400: Invalid type for 'graders.input': expected an object, but got a string instead.

Specification mismatch

In the current public specification, GraderMulti.graders is a oneOf containing individual StringCheck, TextSimilarity, Python, ScoreModel, and LabelModel graders. It therefore describes a single grader object.

The combined-graders guide instead defines graders as an object whose keys name constituent graders. calculate_output refers to those keys. For example:

{
  "type": "multi",
  "name": "example",
  "graders": {
    "compliant": {
      "type": "string_check",
      "name": "compliant",
      "input": "{{sample.output_json.compliant}}",
      "reference": "{{item.compliant}}",
      "operation": "eq"
    }
  },
  "calculate_output": "0.5 * compliant"
}

The generated SDK currently follows the single-grader union, so the source example serializes the grader fields directly under graders and the service treats input as a named grader with an invalid string value.

There is also an inconsistent GraderMulti example in the specification: it uses an array, which matches neither the single-grader schema nor the documented named-map contract.

Expected correction

Describe graders as an object whose additional properties use the supported individual-grader union, and replace the array example with a named map whose keys match calculate_output. Regenerate the published schema and SDK models. Preserve the original SDK reproduction as a validation case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with openapi.yaml around line 41033 and compare the GraderMulti schema with the combined-graders guide. Inspect the inconsistent GraderMulti example and the generated schema and SDK models. Done means the named-map contract and matching example are represented, regeneration completes, and the original SDK reproduction validates the correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.