allegro / allegro/bigflow

BigQuery serialization/deserialization framework for Beam

Aperta
#144 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
120
Fork
23
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

There are a few issues when it comes to reading and writing to BQ in a Beam process.

1. When you read a table, you typically want to map the dictionaries you are getting as input, to an object (which represents your domain). The second thing is that it's nice to have type checking when you deserialize BQ input.

We need a tool that can map a dictionary to an object of the specified dataclass, and can perform type checking.

2. When you write records to a BQ table, you need to map them to the dictionary first. Again, it's nice to have out-of-the-box type checking in this scenario.

We need a tool that can map an object to of the specified dataclass to the dictionary, and can perform type checking.

3. When you write to a BQ table, you need to specify the table schema. We need a tool that can generate BQ schema from the dataclass. Combined with the serializer from pt. 2, we have a single dataclass which describes the final output table.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.