cloudpipe / cloudpipe/cloudpickle

Feature request: cloudpickle.patch_multiprocessing() utility for ForkingPickler replacement

Open
#589 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

## Summary

Propose adding a `cloudpickle.patch_multiprocessing()` helper that replaces `multiprocessing.reduction.ForkingPickler` with a cloudpickle-based pickler, enabling `Pool.map(lambda x: x**2, range(10))` to work out of the box.

## Motivation: ecosystem fragmentation

Every project that needs cloudpickle + `multiprocessing.Pool` independently reinvents this patching. At least 6 projects maintain their own version:

| Project | Approach |
|---------|----------|
| **loky/joblib** | Full custom `_LokyPickler` subsystem in `loky/backend/reduction.py` |
| **PySpark** | Own `CloudPickleSerializer` wrapping `cloudpickle.dumps/loads` |
| **Ray** | Bundled fork as `ray.cloudpickle` with custom object store |
| **Dask** | Custom serialization protocol in distributed scheduler |
| **multiprocess** | Complete fork of CPython's multiprocessing with dill substituted |
| **trading-strategy/exec-sandbox/pypeln/pyrocko** | Ad-hoc monkey patches of varying correctness |

Most ad-hoc implementations are **incomplete** because of a non-obvious CPython pitfall (see below).

## The `_ForkingPickler` double-binding pitfall

CPython has **two separate name bindings** for `ForkingPickler`:

```python
# multiprocessing/reduction.py
class ForkingPickler(pickle.Pickler):
...
```

```python
# multiprocessing/connection.py
from .context import reduction
_ForkingPickler = reduction.ForkingPickler # captured at import time

class Connection:
def send(self, obj):
self._send_bytes(_ForkingPickler.dumps(obj)) # uses the captured reference
```

Patching `reduction.ForkingPickler` alone is **insufficient** — `Connection.send()` still uses the stale `_ForkingPickler` reference captured at import time. You must also patch `multiprocessing.connection._ForkingPickler`. Most ad-hoc implementations miss this.

Additionally, `reduction.dump()` is a module-level function that also needs replacing for completeness.

## Proposed API

```python
import cloudpickle

cloudpickle.patch_multiprocessing()
```

One call, idempotent, patches all three binding sites:
1. `multiprocessing.reduction.ForkingPickler` — the class
2. `multiprocessing.reduction.dump` — the module-level helper
3. `multiprocessing.connection._ForkingPickler` — the import-time captured reference

## Reference implementation

Here's a minimal working implementation (tested on Python 3.14):

```python
import copyreg
import io
import multiprocessing.connection
import multiprocessing.reduction

import cloudpickle

class CloudForkingPickler(cloudpickle.Pickler):
"""ForkingPickler replacement backed by cloudpickle."""
_extra_reducers = {}
_copyreg_dispatch_table = copyreg.dispatch_table

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.dispatch_table = self._copyreg_dispatch_table.copy()
self.dispatch_table.update(self._extra_reducers)

@classmethod
def register(cls, type, reduce):
cls._extra_reducers[type] = reduce

@classmethod
def dumps(cls, obj, protocol=None):
buf = io.BytesIO()
cls(buf, protocol).dump(obj)
return buf.getbuffer()

loads = staticmethod(cloudpickle.loads)

def patch_multiprocessing():
"""Replace multiprocessing's ForkingPickler with cloudpickle-based version."""
# 1. The class itself
multiprocessing.reduction.ForkingPickler = CloudForkingPickler
# 2. The module-level dump() helper
multiprocessing.reduction.dump = lambda obj, file, protocol=None: \
CloudForkingPickler(file, protocol).dump(obj)
# 3. The import-time captured reference in connection.py
multiprocessing.connection._ForkingPickler = CloudForkingPickler
```

After `patch_multiprocessing()`:
```python
from multiprocessing import Pool
with Pool(4) as p:
print(p.map(lambda x: x**2, range(10)))
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
```

## Why cloudpickle (not CPython)

There's an [open discussion on discuss.python.org](https://discuss.python.org/t/add-api-to-change-pickler-for-multiprocessing/30299) about adding a pluggable pickler API to multiprocessing, but no PEP has materialized. cloudpickle is the pragmatic place for this — it already provides `Pickler`/`dumps`/`loads`, and adding a one-shot integration helper is a small, natural extension.

## Alternatives considered

- **"Just use loky/joblib"** — Valid for many users, but loky replaces the entire process management layer. Many projects only need cloudpickle serialization with stdlib `multiprocessing.Pool`.
- **"Just use multiprocess (dill)"** — Requires replacing all `multiprocessing` imports. dill is heavier than cloudpickle and has different serialization semantics.
- **"Document the pattern instead"** — The `_ForkingPickler` double-binding makes documentation insufficient; people will keep getting it wrong.

Happy to submit a PR if there's interest.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.