airbytehq / airbytehq/airbyte

[destination-gcs] JSON 2 Avro converter failing on source schema change

Ouverte
#70,255 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area/connectors autoteam community connectors/destination/gcs needs-triage team/extensibility team/use type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Connector Name

destination-gcs

### Connector Version

0.4.4

### What step the error happened?

During the sync

### Relevant information

This issue arose during a source PostgreSQL and destination GCS connection. The source was configured to ingest data in CDC mode. The source schema had suffered a data type schema change, but there were events to be ingested from both schemas (old and new one). The error obtained was the following:
```
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
This error shows that the JSON to GCS converter of the destination does not support schema changes within the same data block to be ingested. In our view, this should be adapted so that the data type becomes a dynamic one. The user would then have to handle the schema evolution at the destination level.

### Relevant log output

```shell
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```

### Contribute

- [ ] Yes, I want to contribute

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.