airbytehq / airbytehq/airbyte-python-cdk

Improve error messages when json schema validation fails (Pydantic discriminators)

Abierto
#615 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

We get really noising and difficult to parse error messages when manifests don't successfully validate against the json schema.

JSON Schema (and Pydantic) have a feature called "discriminator" which can help with this:

https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions

> When Union validation fails, **error messages can be quite verbose**, as they will produce validation errors for each case in the union. This is especially noticeable when dealing with recursive models, where reasons may be generated at each level of recursion. **Discriminated unions help to simplify error messages in this case, as validation errors are only produced for the case with a matching discriminator value.**

Today: we get a huge error of the input not matching one of n "anyOf" conditions, which could be tens or hundreds of lines of error, without explicitly naming the field that was extra or missing.

Afterwards: we would get an error like `field 'foo' is missing` or `unexpected field 'food' field`

Sample code from the docs:

```py
class Model(BaseModel):
pet: Cat | Dog | Lizard = Field(discriminator='pet_type')
n: int
```

Taking our [`SimpleRetriever.selector`](https://github.com/airbytehq/airbyte/blob/57acc8b84c0b43cc836ab7b79ec02c60e18d2aab/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_manifest_only/declarative_component_schema.py#L1545-L1551) field, for example:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
)
```

We'd update as follows to declare `discriminator`:

```py
class SimpleRetriever(BaseModel):
...
requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API.",
discriminator="type",
)
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.