googleapis / googleapis/synthtool

[Python] stronger / more discoverable discouragement of using multiprocessing module is needed

Open
#902 2 comments 0 reactions 0 assignees View on GitHub
lang: python priority: p3 type: docs
Dominant language
Python
Stars
108
Forks
102
Avg merge
1h 23m
Merged PRs (30d)
1

Description

**Background**

I was helping a customer in https://stackoverflow.com/q/64880779/101923 who was attempting to use the `multiprocessing` module with the BigQuery Storage API. Now that I know to look for it, I see the note in the [client library documentation landing page](https://googleapis.dev/python/bigquerystorage/latest/index.html).

> Because this client uses grpcio library, it is safe to share instances across threads. In multiprocessing scenarios, the best practice is to create client instances after the invocation of os.fork() by multiprocessing.Pool or multiprocessing.Process.

**Problems encountered**

- Note claims that multiprocessing _can_ be done. This is (probably?) accurate, but difficult to follow without an example. Customers have to create all worker pools / processes before creating any client objects. This isn't always feasible. For example, with the BigQuery Storage API, the ideal number of workers depends on the number of streams returned by the session create request.
- I wish there was a link to more information about why grpcio has trouble with multiprocessing.
- This note is awkwardly placed between reference documentation links. It's been in the templates for a few months now and I just recently discovered it.
- This note is not linkable -- would be better on its own page or at least having an anchor tag that can be linked to.

**Proposed solution**

- Add an anchor tag to the note / make a separate page that can be linked to.

Some options:

1. We tell people not to use `multiprocessing` at all and link to known issues.
2. We create some code examples that do use `multiprocessing` successfully. Make sure these are run in system tests to catch regressions in `grpcio`.

**References**

* GRPC note about multiprocessing https://github.com/grpc/grpc/blob/master/doc/fork_support.md (out of date?)
* Enable fork support for Python by default -- https://github.com/grpc/grpc/issues/19158
* Hanging bug with Python fork -- https://github.com/grpc/grpc/issues/18075

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.