googleapis / googleapis/google-cloud-go

bigquery/storage/managedwriter: better documentation for AppendRows usage across goroutines

Open
#8,485 4 comments 1 reaction 1 assignee Claimed by @shollyman View on GitHub
api: bigquery priority: p2 type: docs
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

**Is your feature request related to a problem? Please describe.**
I'm trying to understand how to stream high-throughput data into the `default stream`. Should I be making a new `ManagedStream` for every single request and appending a single row? Should I instead make a single `ManagedStream` for the whole application instance and call `AppendRows` from each request? Is `AppendRows` safe for concurrent use? If it's not safe for concurrent use then should I have a single goroutine that bundles incoming rows and appends them in batches?

**Describe the solution you'd like**
There needs to be more docs on usage of the a `ManagedStream` in different cases and what you should and should not do. The existing [Best Practices](https://cloud.google.com/bigquery/docs/write-api-best-practices) docs do not cover the go client.

**Describe alternatives you've considered**
Alternatives would be just picking a path and seeing if it breaks but I'd be nice to get some guidance ahead of time.

**Additional context**
We have a centralized service that will be handling hundreds or thousands of rows per second and we believe the Storage Write API is the best choice for us. We plan to have a single "leader" instance accepting all rows to append from various sources but it's not obvious how the implementation should work to actually use the `ManagedWriter` at scale.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.