airbytehq / airbytehq/airbyte-python-cdk

Improve error messages when json schema validation fails (Pydantic discriminators)

Aperta
#615 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
26
Fork
53
Merge medio
2g 6h
PR unite (30g)
10

Descrizione

We get really noising and difficult to parse error messages when manifests don't successfully validate against the json schema.

JSON Schema (and Pydantic) have a feature called "discriminator" which can help with this:

https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions

> When Union validation fails, **error messages can be quite verbose**, as they will produce validation errors for each case in the union. This is especially noticeable when dealing with recursive models, where reasons may be generated at each level of recursion. **Discriminated unions help to simplify error messages in this case, as validation errors are only produced for the case with a matching discriminator value.**

Today: we get a huge error of the input not matching one of n "anyOf" conditions, which could be tens or hundreds of lines of error, without explicitly naming the field that was extra or missing.

Afterwards: we would get an error like `field 'foo' is missing` or `unexpected field 'food' field`

Sample code from the docs:

```py
class Model(BaseModel):
pet: Cat | Dog | Lizard = Field(discriminator='pet_type')
n: int
```

Taking our [`SimpleRetriever.selector`](https://github.com/airbytehq/airbyte/blob/57acc8b84c0b43cc836ab7b79ec02c60e18d2aab/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_manifest_only/declarative_component_schema.py#L1545-L1551) field, for example:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
)
```

We'd update as follows to declare `discriminator`:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
discriminator="type",
)
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.