numpy / numpy/numpy

np.memmap works differently on windows and linux/mac

Open
#13,338 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Hi. I have a big array (shapes 40k x 40k and bigger), and I want to use memmap and multiprocessing to fill it. I coded simple example to test it, and it does not work on windows. Expected result is to have values saved by each process (see screenshots).

Any suggestions?

Valid result (tested on virtualboxed xubuntu and MacOS):
obraz

Invalid result - array empty (Windows 7):
obraz

Reproducing code example:
import os
import numpy as np
import time
import multiprocessing as mp


def child(v, lock, m, p):
    np.random.seed(int.from_bytes(os.urandom(4), byteorder='little'))
    while True:
        vval = 0
        with lock:
            if v.value > 0:
                v.value -= 1
                vval = v.value
            else:
                break
        sl = np.random.rand()
        print(f'p{p} sleeping for {sl}')
        time.sleep(sl)
        m[0, vval] = p
        print(f'proc {p}: memmap[{vval}]={p} memmap={m}')



if __name__ == '__main__':
    memmap = np.memmap(
        'test1.memmap',
        dtype='int',
        mode='w+', shape=(1,10)
        )

    value = mp.Value('i', 10)
    lock = mp.Lock()
    procs = [mp.Process(
        target=child,
        args=(value, lock, memmap, i)
        ) for i in range(1,3)]
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()
    
    print(f'after join, result:')
    print(memmap)

Numpy/Python version information:

windows: 1.16.2 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
linux: 1.16.2 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided multiprocessing and np.memmap reproducer on Windows and on Linux or macOS, using the listed Python and NumPy versions. Compare the child-process writes and the final array after joining; done means the platform difference is explained and a reproducible fix or documented limitation is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.