airbytehq / airbytehq/airbyte-python-cdk

Improve error messages when json schema validation fails (Pydantic discriminators)

Open
#615 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
53
Avg merge
2d 6h
Merged PRs (30d)
10

Description

We get really noising and difficult to parse error messages when manifests don't successfully validate against the json schema.

JSON Schema (and Pydantic) have a feature called "discriminator" which can help with this:

https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions

> When Union validation fails, **error messages can be quite verbose**, as they will produce validation errors for each case in the union. This is especially noticeable when dealing with recursive models, where reasons may be generated at each level of recursion. **Discriminated unions help to simplify error messages in this case, as validation errors are only produced for the case with a matching discriminator value.**

Today: we get a huge error of the input not matching one of n "anyOf" conditions, which could be tens or hundreds of lines of error, without explicitly naming the field that was extra or missing.

Afterwards: we would get an error like `field 'foo' is missing` or `unexpected field 'food' field`

Sample code from the docs:

```py
class Model(BaseModel):
pet: Cat | Dog | Lizard = Field(discriminator='pet_type')
n: int
```

Taking our [`SimpleRetriever.selector`](https://github.com/airbytehq/airbyte/blob/57acc8b84c0b43cc836ab7b79ec02c60e18d2aab/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_manifest_only/declarative_component_schema.py#L1545-L1551) field, for example:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
)
```

We'd update as follows to declare `discriminator`:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
discriminator="type",
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.