python / python/cpython

Can't use XMLRPC errors with multiprocessing and pickle

Open
#93,542 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

Can't use XMLRPC errors with multiprocessing and pickle modules. Multiprocessing freezes.

Bug example

from multiprocessing import Pool
from xmlrpc.client import ProtocolError


def raise_error(_):
    raise ProtocolError("1", 2, "3", {})


Pool(1).map(raise_error, (["A", 5]))

Expected output

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\ppatrin\Projects\test.py", line 6, in raise_error
    raise ProtocolError("1", 2, "3", {})
xmlrpc.client.ProtocolError: <ProtocolError for 1: 2 3>

Actual output

"C:\Program Files\Python310\python.exe" C:/Users/ppatrin/Projects/test.py
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python310\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 576, in _handle_results
    task = get()
  File "C:\Program Files\Python310\lib\multiprocessing\connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: ProtocolError.__init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'

And multiprocessing processes frozes.

Real problem

import pickle
from xmlrpc.client import ProtocolError

pickle.loads(pickle.dumps(ProtocolError(1, "2", 3, {})))

>>> TypeError: __init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'

How to fix

We should fix a super call in exception method initializer.

class ProtocolError(Error):
    """Indicates an HTTP protocol error."""
    def __init__(self, url, errcode, errmsg, headers):
---        Error.__init__(self)
+++        Error.__init__(self, url, errcode, errmsg, headers)
        self.url = url
        self.errcode = errcode
        self.errmsg = errmsg
        self.headers = headers

Your environment

  • CPython versions tested on: 3.10.4
  • Operating system and architecture: Windows 11 and Ubuntu 20.04.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ProtocolError initializer in the xmlrpc.client module and reproduce the pickle round-trip shown in the issue. The work is done when ProtocolError can be pickled and unpickled successfully and the multiprocessing example no longer freezes while returning the expected error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.