[destination-gcs] JSON 2 Avro converter failing on source schema change
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Connector Name
destination-gcs
### Connector Version
0.4.4
### What step the error happened?
During the sync
### Relevant information
This issue arose during a source PostgreSQL and destination GCS connection. The source was configured to ingest data in CDC mode. The source schema had suffered a data type schema change, but there were events to be ingested from both schemas (old and new one). The error obtained was the following:
```
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
This error shows that the JSON to GCS converter of the destination does not support schema changes within the same data block to be ingested. In our view, this should be adapted so that the data type becomes a dynamic one. The user would then have to handle the schema evolution at the destination level.
### Relevant log output
```shell
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
### Contribute
- [ ] Yes, I want to contribute
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.