openai / openai/openai-openapi
bug in YAML in property testing_criteria of type Eval
Open
Nobody has claimed this yet.
bug
specification
- Dominant language
- No language data
- Stars
- 2.5k
- Forks
- 527
- Avg merge
- 1h 46m
- Merged PRs (30d)
- 2
Description
In
Eval:
type: object
title: Eval
description: |
An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o3-mini is better at my usecase than gpt-4o
properties:
object:
type: string
enum:
- eval
default: eval
description: The object type.
x-stainless-const: true
id:
type: string
description: Unique identifier for the evaluation.
name:
type: string
description: The name of the evaluation.
example: Chatbot effectiveness Evaluation
data_source_config:
type: object
description: Configuration of data sources used in runs of the evaluation.
oneOf:
- $ref: "#/components/schemas/EvalCustomDataSourceConfig"
- $ref: "#/components/schemas/EvalStoredCompletionsDataSourceConfig"
testing_criteria:
default: eval
description: A list of testing criteria.
type: array
items:
oneOf:
- $ref: "#/components/schemas/EvalLabelModelGrader"
- $ref: "#/components/schemas/EvalStringCheckGrader"
- $ref: "#/components/schemas/EvalTextSimilarityGrader"
- $ref: "#/components/schemas/EvalPythonGrader"
- $ref: "#/components/schemas/EvalScoreModelGrader"
default: eval seems like a bug:
testing_criteria:
default: eval
description: A list of testing criteria.
type: array
items:
oneOf:
- $ref: "#/components/schemas/EvalLabelModelGrader"
- $ref: "#/components/schemas/EvalStringCheckGrader"
- $ref: "#/components/schemas/EvalTextSimilarityGrader"
- $ref: "#/components/schemas/EvalPythonGrader"
- $ref: "#/components/schemas/EvalScoreModelGrader"
The property is of array type, and the array could contain one of the nominated types. So what is default value "eval"?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the Eval schema in the OpenAPI YAML and inspect the testing_criteria property against its array type and listed oneOf items. Validate the specification after the correction; done means testing_criteria no longer declares an invalid default and the document remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- openapi
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100