BigQuery serialization/deserialization framework for Beam
- 主要語言
- Python
- 星號
- 120
- 分支
- 23
- PR 合併指標
- 30 天內沒有已合併 PR
描述
There are a few issues when it comes to reading and writing to BQ in a Beam process.
1. When you read a table, you typically want to map the dictionaries you are getting as input, to an object (which represents your domain). The second thing is that it's nice to have type checking when you deserialize BQ input.
We need a tool that can map a dictionary to an object of the specified dataclass, and can perform type checking.
2. When you write records to a BQ table, you need to map them to the dictionary first. Again, it's nice to have out-of-the-box type checking in this scenario.
We need a tool that can map an object to of the specified dataclass to the dictionary, and can perform type checking.
3. When you write to a BQ table, you need to specify the table schema. We need a tool that can generate BQ schema from the dataclass. Combined with the serializer from pt. 2, we have a single dataclass which describes the final output table.
貢獻指南
這個儲存庫沒有索引到貢獻指南
評估
這個 Issue 還沒有評估資料。