Supporting out-of-band buffers with pickle protocol 5
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## Feature description
Typically pickling in Python creates a large `bytes` object with types, functions, and data all packed in to allow easy reconstruction later. Originally pickling was focused on reading/writing to disk. However these days it is increasingly using as a serialization protocol for objects on the wire. In this case the copies of data required to put everything in a single `bytes` object hurts performance and doesn't offer much (as the data could be shipped along in separate buffers without copying).
For these reasons, Python added support for [out-of-band buffers in pickle]( https://docs.python.org/3/library/pickle.html#out-of-band-buffers ), which allows the user to flag buffers of data for pickle to extract and send alongside the typical `bytes` object (thus avoiding unneeded copying of data). This was submitted and accepted as [PEP 574]( https://www.python.org/dev/peps/pep-0574/ ) and is part of Python 3.8 (along with [a backport package for Python 3.5, 3.6, and 3.7]( https://github.com/pitrou/pickle5-backport )). On the implementation side this just comes down to implementing `__reduce_ex__` instead of `__reduce__` (basically the same with a `protocol` version argument) and placing any `bytes`-like data (like NumPy arrays and `memoryview`s) into [`PickleBuffer` objects]( https://docs.python.org/3/library/pickle.html#pickle.PickleBuffer ). For older pickle protocols this step can simply be skipped. Here's [an example]( https://docs.python.org/3/library/pickle.html#example ). The rest is on libraries using protocol 5 (like Dask) to implement and use.
## Could the feature be a [custom component](https://spacy.io/usage/processing-pipelines#custom-components) or [spaCy plugin](https://spacy.io/universe)?
If so, we will tag it as [`project idea`](https://github.com/explosion/spaCy/labels/project%20idea) so other users can take it on.
I don't think so as this relies on changing the pickle implementations of spaCy objects. Though I could be wrong :)
Contributor guide
Assessment
This issue has not been assessed yet.