googleapis / googleapis/google-cloud-go

bigquery: storage writer API: Guidance on multiplexing

Open
#11,902 1 comment 0 reactions 2 assignees Claimed by @shollyman View on GitHub
api: bigquery type: question
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Is your feature request related to a problem? Please describe.

As others have mentioned (for example, [here](https://github.com/googleapis/google-cloud-go/issues/6033) ), while the connection multiplexing seems very promising, it is difficult to figure out how to use it appropriately and what tuning should be done, if any.

We are planning an architecture with thousands of datasets, about 50 tables per dataset, and 5000 inserts/sec across all of these 50,000 tables (not dissimilar from what others have described in other issues).

However, in our initial experiments it has been challenging to achieve low write latency at any scale.

We are using one client connection per pod, creating new managed streams on demand for each dataset+table combination (as required). We cache and re-use the managed streams. (Obviously in a production environment we'd use multiple pods, but we're just investigating latency issues for now.)

## Describe the solution you'd like

It would be extremely helpful if there was better documentation about how to effectively use and tune multiplexing.

1. how many streams can be multiplexed per connection? Or can it be treated as unlimited?
1. is caching managed streams the correct strategy?
1. the documentation talks about the need to handle cold start with application streams, but we have been finding cold start a problem even with the default stream (that is, creating managed streams seems to be serialized and to take non-trivial time)
1. is multiplexing affected by distribution of tables per dataset? (I sort of assume not, but lots of tables, fewer datasets, or fewer tables, lots of datasets?)
1. any guidance on the various tuning options? `WithMultiplexPoolLimit`, `WithGRPCConnectionPool`, `WithDefaultInflightRequests`, `WithDefaultInflightBytes`? There are also flow control options, but it appears no flow control is done unless those are set?
1. do other variables come into play? BigQuery row size, partitioning?

## Describe alternatives you've considered

We haven't really. We think multiplexing should theoretically work well for us but we're trying to figure out how much work we need to do to distribute streams across connections and possibly across pods (which will then require some PubSub message affinity work, etc)

## Additional context

https://github.com/googleapis/google-cloud-go/issues/6033

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.