googleapis / googleapis/google-cloud-go

bigquery/storage/managedwriter: managedwriter should be compatible with protoc-gen-bq-schema

Open
#7,766 7 comments 2 reactions 1 assignee Claimed by @shollyman View on GitHub
api: bigquery type: question
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

**Is your feature request related to a problem? Please describe.**
It seems that when a protobuf is used to generate a BigQuery schema using https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema, the schema generated is incompatible with the managedwriter stream that can be created via `adapt.NormalizeSchema()`.

This makes `managedwriter` less useful in being able to write to a BQ table created using `protoc-gen-bq-schema`.

This is because there is currently a difference in how proto fields are converted to bigquery fields in `protoc-gen-bq-schema` vs in `managedwriter`.

`protoc-gen-bq-schema` converts `google.protobuf.Duration` to `STRING` (https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema/blob/master/pkg/converter/convert.go#L20), however `managedwriter/protoconversion` only handles a subset and doesn't handle Duration or Timestamp properly (https://github.com/googleapis/google-cloud-go/blob/main/bigquery/storage/managedwriter/adapt/protoconversion.go#L89)

**Describe the solution you'd like**
What is needed here is to bring feature parity between the BQ schema and the `protoconversion.go` module, which means adding/changing support for the following translations:
```
".google.protobuf.Duration" --> "STRING",
".google.protobuf.Timestamp" --> "TIMESTAMP",
```

**Describe alternatives you've considered**
I do not know of any other ideas.

**Additional context**
We generate a very very large BQ schema (impossible to generate by hand) from a very well-used proto. This makes it very difficult to write without specifying the schema or dropping many of the fields.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.