BigQuery serialization/deserialization framework for Beam
- Lingua principale
- Python
- Stelle
- 120
- Fork
- 23
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
There are a few issues when it comes to reading and writing to BQ in a Beam process.
1. When you read a table, you typically want to map the dictionaries you are getting as input, to an object (which represents your domain). The second thing is that it's nice to have type checking when you deserialize BQ input.
We need a tool that can map a dictionary to an object of the specified dataclass, and can perform type checking.
2. When you write records to a BQ table, you need to map them to the dictionary first. Again, it's nice to have out-of-the-box type checking in this scenario.
We need a tool that can map an object to of the specified dataclass to the dictionary, and can perform type checking.
3. When you write to a BQ table, you need to specify the table schema. We need a tool that can generate BQ schema from the dataclass. Combined with the serializer from pt. 2, we have a single dataclass which describes the final output table.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.