airbytehq / airbytehq/airbyte

Destination S3, S3-Glue: Add option to convert objects to strings

未關閉
#27,171 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors community connectors/destination/s3 connectors/destination/s3-glue connectors/destinations-database frozen team/destinations type/enhancement
主要語言
Python
星號
22.1k
分支
5.3k
PR 合併指標
PR 指標待擷取

描述

### What area the feature impact?

Connectors

### Revelant Information

# Description/Context
Airbyte has a representation of data types that all source connectors map their own types into, and which the destination connectors need to map from into their own types ([reference](https://docs.airbyte.com/understanding-airbyte/supported-data-types/)). Nested objects in the source data are represented as an `Object`.

The S3 and S3-Glue connectors are relying on the upstream [Apache Hive JsonSerDe](https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java#L53) library to handle serialization of JSON formatted data. By default, items being deserialized are expected to be wrapped in Hadoop Writable objects and objects being serialized are expected to be Java primitive objects.

The issue is that when nested documents are being serialized as an `object`, the data is escaped incorrectly and the resulting record in the Hive table is always `{`. We need to create a configurable option that enables the S3 and S3-Glue destination connectors to serialize any root level `object` as a `string` (including _airbyte_data).

# Plan/Design
- [ ] Add a way to specify whether to enable the "stringify" feature for S3 and S3-Glue destinations with JSON Lines output
- [ ] Preserve existing functionality and prevent breaking current implementations by setting the default behavior to treat `object` types as `object` ( Options should be `Default` or `Stringify`)
- [ ] Extend or refactor the existing configs to thread through the stringify argument
- [ ] Update the JsonLSerializedBuffer logic to convert root level objects to strings with the given input parameter
- [ ] Add tests as needed

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。