np.memmap works differently on windows and linux/mac
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Hi. I have a big array (shapes 40k x 40k and bigger), and I want to use memmap and multiprocessing to fill it. I coded simple example to test it, and it does not work on windows. Expected result is to have values saved by each process (see screenshots).
Any suggestions?
Valid result (tested on virtualboxed xubuntu and MacOS):

Invalid result - array empty (Windows 7):

Reproducing code example:
import os
import numpy as np
import time
import multiprocessing as mp
def child(v, lock, m, p):
np.random.seed(int.from_bytes(os.urandom(4), byteorder='little'))
while True:
vval = 0
with lock:
if v.value > 0:
v.value -= 1
vval = v.value
else:
break
sl = np.random.rand()
print(f'p{p} sleeping for {sl}')
time.sleep(sl)
m[0, vval] = p
print(f'proc {p}: memmap[{vval}]={p} memmap={m}')
if __name__ == '__main__':
memmap = np.memmap(
'test1.memmap',
dtype='int',
mode='w+', shape=(1,10)
)
value = mp.Value('i', 10)
lock = mp.Lock()
procs = [mp.Process(
target=child,
args=(value, lock, memmap, i)
) for i in range(1,3)]
for proc in procs:
proc.start()
for proc in procs:
proc.join()
print(f'after join, result:')
print(memmap)
Numpy/Python version information:
windows: 1.16.2 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
linux: 1.16.2 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided multiprocessing and np.memmap reproducer on Windows and on Linux or macOS, using the listed Python and NumPy versions. Compare the child-process writes and the final array after joining; done means the platform difference is explained and a reproducible fix or documented limitation is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100