Can't use XMLRPC errors with multiprocessing and pickle
Open
Nobody has claimed this yet.
topic-multiprocessing
type-bug
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- PR merge metrics
- PR metrics pending
Description
Can't use XMLRPC errors with multiprocessing and pickle modules. Multiprocessing freezes.
Bug example
from multiprocessing import Pool
from xmlrpc.client import ProtocolError
def raise_error(_):
raise ProtocolError("1", 2, "3", {})
Pool(1).map(raise_error, (["A", 5]))
Expected output
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "C:\Users\ppatrin\Projects\test.py", line 6, in raise_error
raise ProtocolError("1", 2, "3", {})
xmlrpc.client.ProtocolError: <ProtocolError for 1: 2 3>
Actual output
"C:\Program Files\Python310\python.exe" C:/Users/ppatrin/Projects/test.py
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\threading.py", line 1009, in _bootstrap_inner
self.run()
File "C:\Program Files\Python310\lib\threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 576, in _handle_results
task = get()
File "C:\Program Files\Python310\lib\multiprocessing\connection.py", line 256, in recv
return _ForkingPickler.loads(buf.getbuffer())
TypeError: ProtocolError.__init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'
And multiprocessing processes frozes.
Real problem
import pickle
from xmlrpc.client import ProtocolError
pickle.loads(pickle.dumps(ProtocolError(1, "2", 3, {})))
>>> TypeError: __init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'
How to fix
We should fix a super call in exception method initializer.
class ProtocolError(Error):
"""Indicates an HTTP protocol error."""
def __init__(self, url, errcode, errmsg, headers):
--- Error.__init__(self)
+++ Error.__init__(self, url, errcode, errmsg, headers)
self.url = url
self.errcode = errcode
self.errmsg = errmsg
self.headers = headers
Your environment
- CPython versions tested on: 3.10.4
- Operating system and architecture: Windows 11 and Ubuntu 20.04.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the ProtocolError initializer in the xmlrpc.client module and reproduce the pickle round-trip shown in the issue. The work is done when ProtocolError can be pickled and unpickled successfully and the multiprocessing example no longer freezes while returning the expected error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100