python / python/cpython

Weirdo multiprocessing: Shared objects taking more time in sharing smaller data than larger data between multiple processes.

未关闭
#126,471 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

extension-modules pending topic-multiprocessing type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

server.py

from multiprocessing.managers import BaseManager
from queue import Queue
queue = Queue()
class QueueManager(BaseManager): pass
QueueManager.register('get_queue', callable=lambda:queue)
m = QueueManager(address=('', 5000), authkey=b'abracadabra')
s = m.get_server()
s.serve_forever()

consumer.py

import time

from multiprocessing.managers import BaseManager

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')
m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

while True:
    t = time.time()
    x = queue.get()

producerA.py


from multiprocessing.managers import BaseManager
import time
import numpy as np

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')

m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

out_img = np.zeros((128, 128, 3), dtype=np.uint8)

for i in range(100):
    t = time.time()
    queue.put(
    {
        'type' : 'not working',
        'data': out_img
    }    
    )
    print('put took', (time.time() - t)*1000)

producerB.py

from multiprocessing.managers import BaseManager
import time
import numpy as np

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')

m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

out_img = np.zeros((256, 256, 3), dtype=np.uint8)

for i in range(100):
    t = time.time()
    queue.put(
    {
        'type' : 'not working',
        'data': out_img
    }    
    )
    print('put took', (time.time() - t)*1000)

steps to reproduce the issue

  • run server.py
  • run consumer.py
  • run producerA.py and check the put time and compare it with producerB.py put time.

put time in producerA.py is higher than producerB.py, however the size of data being send through shared queue objects is more in producerB.py( 256x256 ) than in producerA.py (128x128).
I don't have much context if this is related to numpy or cpython multiprocessing, but this is definitely shouldn't be the case.
Ideally it should take more time in producerB.py beacuase the size of data is relatively higher relative to producerA.py.

This is my first bug report, I'm hoping to learn something...

Thanks everyone...

CPython versions tested on:

3.10

Operating systems tested on:

Linux

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先运行 server.py 和 consumer.py,然后在所报告的 Linux 和 CPython 3.10 环境中比较 producerA.py 和 producerB.py 的耗时。跟踪 CPython 的 multiprocessing manager queue 和 NumPy 序列化路径中的行为。报告没有指定源文件或测试目标,因此完成此项工作需要确认原因并提出适合项目的解决方案。

由索引模型根据 Issue 内容生成。

评估

技术栈
numpy, python
领域
distributed-systems
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。