python / python/cpython

Weirdo multiprocessing: Shared objects taking more time in sharing smaller data than larger data between multiple processes.

Open
#126,471 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules pending topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

server.py

from multiprocessing.managers import BaseManager
from queue import Queue
queue = Queue()
class QueueManager(BaseManager): pass
QueueManager.register('get_queue', callable=lambda:queue)
m = QueueManager(address=('', 5000), authkey=b'abracadabra')
s = m.get_server()
s.serve_forever()

consumer.py

import time

from multiprocessing.managers import BaseManager

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')
m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

while True:
    t = time.time()
    x = queue.get()

producerA.py


from multiprocessing.managers import BaseManager
import time
import numpy as np

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')

m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

out_img = np.zeros((128, 128, 3), dtype=np.uint8)

for i in range(100):
    t = time.time()
    queue.put(
    {
        'type' : 'not working',
        'data': out_img
    }    
    )
    print('put took', (time.time() - t)*1000)

producerB.py

from multiprocessing.managers import BaseManager
import time
import numpy as np

class QueueManager(BaseManager): pass

QueueManager.register('get_queue')

m = QueueManager(address=('', 5000), authkey=b'abracadabra')
m.connect()
queue = m.get_queue()

out_img = np.zeros((256, 256, 3), dtype=np.uint8)

for i in range(100):
    t = time.time()
    queue.put(
    {
        'type' : 'not working',
        'data': out_img
    }    
    )
    print('put took', (time.time() - t)*1000)

steps to reproduce the issue

  • run server.py
  • run consumer.py
  • run producerA.py and check the put time and compare it with producerB.py put time.

put time in producerA.py is higher than producerB.py, however the size of data being send through shared queue objects is more in producerB.py( 256x256 ) than in producerA.py (128x128).
I don't have much context if this is related to numpy or cpython multiprocessing, but this is definitely shouldn't be the case.
Ideally it should take more time in producerB.py beacuase the size of data is relatively higher relative to producerA.py.

This is my first bug report, I'm hoping to learn something...

Thanks everyone...

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running server.py and consumer.py, then compare the timings from producerA.py and producerB.py on the reported Linux and CPython 3.10 setup. Trace the behavior through CPython's multiprocessing manager queue and NumPy serialization paths. The report names no source file or test target, so completion requires a confirmed cause and a project-appropriate resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.