googleapis / googleapis/google-cloud-rust
BQ Write: `JsonStreamWriter`
Open
api: bigquery
- Dominant language
- Rust
- Stars
- 955
- Forks
- 144
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 279
Description
Implement `JsonStreamWriter` for the BigQuery Storage Write API to allow users to stream JSON records natively.
JSON streaming requires dynamic protobuf translation. The Storage Write API expects encoded protobuf arrays (`ProtoRows`) according to a `ProtoSchema`. Since schemas are fetched at runtime, we cannot rely on `protoc`.
Our implementation is as follow:
1. Fetch the runtime BigQuery `TableSchema`.
2. Translate `TableSchema` into a protobuf AST `DescriptorProto`.
3. Use `prost_reflect` to create dynamic protobuf messages.
4. Unmarshall user-provided JSON onto the schema, encode to bytes and stream it.
Contributor guide
Assessment
This issue has not been assessed yet.