allegro / allegro/bigflow

BigQuery serialization/deserialization framework for Beam

未關閉
#144 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
120
分支
23
PR 合併指標
30 天內沒有已合併 PR

描述

There are a few issues when it comes to reading and writing to BQ in a Beam process.

1. When you read a table, you typically want to map the dictionaries you are getting as input, to an object (which represents your domain). The second thing is that it's nice to have type checking when you deserialize BQ input.

We need a tool that can map a dictionary to an object of the specified dataclass, and can perform type checking.

2. When you write records to a BQ table, you need to map them to the dictionary first. Again, it's nice to have out-of-the-box type checking in this scenario.

We need a tool that can map an object to of the specified dataclass to the dictionary, and can perform type checking.

3. When you write to a BQ table, you need to specify the table schema. We need a tool that can generate BQ schema from the dataclass. Combined with the serializer from pt. 2, we have a single dataclass which describes the final output table.

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。