googleapis / googleapis/google-cloud-python

reat: Bi-Directional Streaming APIs could use BidiRpc base class by default

Open
#16,450 2 comments 0 reactions 1 assignee Claimed by @tswast View on GitHub
priority: p3 type: feature request
Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
3d 4h
Merged PRs (30d)
122

Description

**Is your feature request related to a problem? Please describe.**

I'm trying to implement some fixes for the BigQuery Storage API Write client in https://github.com/googleapis/python-bigquery-storage/pull/278 so that it's easier to use from a multi-threaded environment.

The current interface for bi-directional streaming APIs takes an iterable of requests and returns an iterable of responses. For long-running streams, it's on the consumer of this API to ensure they have a background thread consuming the responses as they write requests to the the input iterable (e.g. via a generator).

It would be preferable if the API instead provided a way to open the stream with an initial request and then subsequently call send/recv on the stream (this it the interface provided by [BidiRpc](https://github.com/googleapis/python-api-core/blob/40f52bf1100cf56b7f9af267d210b8a72fc34f08/google/api_core/bidi.py#L206)

Better yet, if the API guarantees one response message per request message, then having `send` return a future object would an even more natural experience. (This is the case for the API which I'm trying to wrap).

Better further still would be that the generated client could handle flow control, heartbeats, and stream resumption as is done manually in Pub/Sub, Pub/Sub Lite, and Firestore.

**Describe the solution you'd like**

At the very least, automatically wrap Bi-Directional streaming APIs in the BidiRpc class, for a more natural streaming API when working with multiple threads (which you basically have to do unless you somehow interlace iterating over the response iterator while you're generating requests).

**Describe alternatives you've considered**

Status quo means we basically have to make a manual client whenever an API has a Bi-Directional streaming method.

**Additional context**

* Firestore: https://github.com/googleapis/python-firestore/blob/e57258c51e4b4aa664cc927454056412756fc7ac/google/cloud/firestore_v1/watch.py
* Pub/Sub: https://github.com/googleapis/python-pubsub/blob/e907f6e05f59f64a3b08df3304e92ec960997be6/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
* Pub/Sub Lite: https://github.com/googleapis/python-pubsublite/blob/670c4b0e5e5edfa402ab65f455f7cc798d064a77/google/cloud/pubsublite/internal/wire/subscriber_impl.py -- Doesn't actually use the BidiRpc base class because they chose to build an async manual client.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.