airbytehq / airbytehq/airbyte

[destination-gcs] JSON 2 Avro converter failing on source schema change

未關閉
#70,255 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/destination/gcs needs-triage team/extensibility team/use type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

destination-gcs

### Connector Version

0.4.4

### What step the error happened?

During the sync

### Relevant information

This issue arose during a source PostgreSQL and destination GCS connection. The source was configured to ingest data in CDC mode. The source schema had suffered a data type schema change, but there were events to be ingested from both schemas (old and new one). The error obtained was the following:
```
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```
This error shows that the JSON to GCS converter of the destination does not support schema changes within the same data block to be ingested. In our view, this should be adapted so that the data type becomes a dynamic one. The user would then have to handle the schema evolution at the destination level.

### Relevant log output

```shell
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field col1 is expected to be one of these: NULL, LONG. If this is a complex type, check if offending field (path: col1) adheres to schema: a
```

### Contribute

- [ ] Yes, I want to contribute

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。