googleapis / googleapis/google-cloud-go
bigquery/storage/managedwriter: managedwriter should be compatible with protoc-gen-bq-schema
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
**Is your feature request related to a problem? Please describe.**
It seems that when a protobuf is used to generate a BigQuery schema using https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema, the schema generated is incompatible with the managedwriter stream that can be created via `adapt.NormalizeSchema()`.
This makes `managedwriter` less useful in being able to write to a BQ table created using `protoc-gen-bq-schema`.
This is because there is currently a difference in how proto fields are converted to bigquery fields in `protoc-gen-bq-schema` vs in `managedwriter`.
`protoc-gen-bq-schema` converts `google.protobuf.Duration` to `STRING` (https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema/blob/master/pkg/converter/convert.go#L20), however `managedwriter/protoconversion` only handles a subset and doesn't handle Duration or Timestamp properly (https://github.com/googleapis/google-cloud-go/blob/main/bigquery/storage/managedwriter/adapt/protoconversion.go#L89)
**Describe the solution you'd like**
What is needed here is to bring feature parity between the BQ schema and the `protoconversion.go` module, which means adding/changing support for the following translations:
```
".google.protobuf.Duration" --> "STRING",
".google.protobuf.Timestamp" --> "TIMESTAMP",
```
**Describe alternatives you've considered**
I do not know of any other ideas.
**Additional context**
We generate a very very large BQ schema (impossible to generate by hand) from a very well-used proto. This makes it very difficult to write without specifying the schema or dropping many of the fields.
Contributor guide
Assessment
This issue has not been assessed yet.