Can't use XMLRPC errors with multiprocessing and pickle
未关闭
还没有人认领这个 Issue。
topic-multiprocessing
type-bug
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 36k
- 平均合并
- 1 天 9 小时
- 30 天内合并 PR
- 558
描述
Can't use XMLRPC errors with multiprocessing and pickle modules. Multiprocessing freezes.
Bug example
from multiprocessing import Pool
from xmlrpc.client import ProtocolError
def raise_error(_):
raise ProtocolError("1", 2, "3", {})
Pool(1).map(raise_error, (["A", 5]))
Expected output
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "C:\Users\ppatrin\Projects\test.py", line 6, in raise_error
raise ProtocolError("1", 2, "3", {})
xmlrpc.client.ProtocolError: <ProtocolError for 1: 2 3>
Actual output
"C:\Program Files\Python310\python.exe" C:/Users/ppatrin/Projects/test.py
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\threading.py", line 1009, in _bootstrap_inner
self.run()
File "C:\Program Files\Python310\lib\threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "C:\Program Files\Python310\lib\multiprocessing\pool.py", line 576, in _handle_results
task = get()
File "C:\Program Files\Python310\lib\multiprocessing\connection.py", line 256, in recv
return _ForkingPickler.loads(buf.getbuffer())
TypeError: ProtocolError.__init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'
And multiprocessing processes frozes.
Real problem
import pickle
from xmlrpc.client import ProtocolError
pickle.loads(pickle.dumps(ProtocolError(1, "2", 3, {})))
>>> TypeError: __init__() missing 4 required positional arguments: 'url', 'errcode', 'errmsg', and 'headers'
How to fix
We should fix a super call in exception method initializer.
class ProtocolError(Error):
"""Indicates an HTTP protocol error."""
def __init__(self, url, errcode, errmsg, headers):
--- Error.__init__(self)
+++ Error.__init__(self, url, errcode, errmsg, headers)
self.url = url
self.errcode = errcode
self.errmsg = errmsg
self.headers = headers
Your environment
- CPython versions tested on: 3.10.4
- Operating system and architecture: Windows 11 and Ubuntu 20.04.
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 xmlrpc.client 模块中的 ProtocolError 初始化器开始,复现 issue 中展示的 pickle 往返过程。当 ProtocolError 可以成功进行 pickle 和 unpickle,并且 multiprocessing 示例在返回预期错误时不再冻结时,工作就完成了。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- api, backend
- Issue 类型
- 缺陷
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 38/100