[destination-gcs] JSON 2 Avro converter failing on source schema change
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Connector Name
destination-gcs
### Connector Version
0.4.4
### What step the error happened?
During the sync
### Relevant information
This issue arose during a source PostgreSQL and destination GCS connection. The source was configured to ingest data in CDC mode. The source schema had suffered a data type schema change, but there were events to be ingested from both schemas (old and new one). The error obtained was the following:
```
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
This error shows that the JSON to GCS converter of the destination does not support schema changes within the same data block to be ingested. In our view, this should be adapted so that the data type becomes a dynamic one. The user would then have to handle the schema evolution at the destination level.
### Relevant log output
```shell
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
### Contribute
- [ ] Yes, I want to contribute
貢獻指南
評估
這個 Issue 還沒有評估資料。