matplotlib / matplotlib/matplotlib

[Bug]: Matplotlib savefig and close break multiprocessing and numpy thread allocation

Open
#29,733 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
23.2k
Forks
8.5k
Avg merge
1d 6h
Merged PRs (30d)
66

Description

### Bug summary

I tried reporting this as a python bug but they said they don't really deal with bugs depending on third party packages, so I am trying here. I recently updated my python environment and discovered a weird bug that occurs between matplotlib, numpy, and multiprocessing.

Code to demonstrate the bug below. In the code I am running a multiprocessing pool, with a function that multiplies small matrices a large number of times. If run in a fresh environment (from commmand line or in new ipython session) the loops runs once quickly, utilizing all cpu threads. Then a plot is generated, saved and closed, and the loop runs again, but this time only 2 threads are utilized at 100%, and the execution time is much slower.

Time is used only to demonstrate that the code runs much slower when the bug is occurring, on my main machine (48 cores/96 threads) it runs about 30 times slower when the bug occurs. I tested on another machine I have access to with 8 cores/16 threads, the bugged version runs about 10 times slower.

The try/except is used just so the pool is closed properly if you keyboard interrupt.

### Code for reproduction

```Python
import numpy as np
import multiprocessing as mp
from matplotlib import pyplot as plt
import time

def test_function(var):
test_mat=np.zeros((3,3))
for i in range(10**6):
np.matmul(test_mat,test_mat)
return()


def main():
print('starting')

run_multiplier=2
num_threads=mp.cpu_count()#//2
pool=mp.Pool(num_threads)

try:
completed=0
tic=time.time()
for item in pool.imap(test_function,range(num_threads*run_multiplier)):
completed+=1

toc=time.time()-tic
print('Elapsed time without bug %f seconds'%toc)
pool.close()
pool.join()
except:
pool.terminate()
pool.join()

test_array=np.arange(10)
fig=plt.figure()
plt.plot(test_array,test_array,'.')
plt.savefig('test_fig.png')
#plt.close('all')
plt.close(fig)
# plt.close()

pool=mp.Pool(num_threads)
try:
completed=0
tic=time.time()
for item in pool.imap(test_function,range(num_threads*run_multiplier)):
completed+=1

toc=time.time()-tic
print('Elapsed time with bug %f seconds'%toc)
pool.close()
pool.join()
except:
pool.terminate()
pool.join()


if __name__=='__main__':
main()
```

### Actual outcome

When bug occurs 2 threads are run at 100%, the rest have low utilization.

![Image](https://github.com/user-attachments/assets/696d1e38-b638-45f5-a519-a7adcebf0ee9)

### Expected outcome

When bug does not occur all threads are near 100% utilization until the loop completes.

![Image](https://github.com/user-attachments/assets/16621385-befe-42d4-9007-40d7838f37bd)

### Additional information

The bug occurs when either plt.savefig or plt.close are called, although I tested a few different versions of matplotlib, and the behavior changes slightly. With python 12 and matplotlib 3.10.0 the bug occurs with plt.close(fig) and plt.close('all'). With matplotlib 3.9.2 and 3.8.0 the bug does not occur with plt.close('all') but still occurs with plt.close(fig). The bug does not occur with a plain plt.close() in my testing.

This bug also occurs on python 11 with matplotlib 3.8.0, I didn't test other versions. The bug does NOT occur on python 10 for matplotlib 3.7 or 3.10. The bug also does not occur for python 3.8, matplotlib version 3.3.

The bug only occurs when the multiprocessing pool is generated using fork, not spawn (fork is only default on linux, and not available on windows). This suggests matplotlib possibly has a memory leak that is getting copied in the fork, although I also think there must be improper memory access in either multiprocessing or numpy to be reading the memory leaked by matplotlib. Alternatively some kind of threading/multiprocessing has been introduced for matplotlib in newer python version (>=3.11), but it is clashing with the thread allocation in numpy and multiprocessing.

If this is tested in an ipython environment, you will need to restart the kernel. The bug occurs if any figure has ever been saved or closed (depending on the close method) within the ipython session, so if you don't restart the session then the bugged version will just be running twice.

The bug appears to be independent of the backend chosen for matplotlib, I tried qt, tkinter, ipympl, and Agg.

Both computers I have experienced this bug on are running RHEL 9. When the code is run on WSL running Ubuntu the bug does not occur. Apparently this is somehow distribution dependent.

### Operating system

Red Hat Enterprise Linux 9

### Matplotlib Version

3.8, 3.9, and 3.10

### Matplotlib Backend

any

### Python version

3.11 and 3.12

### Jupyter version

n/a

### Installation

conda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction script in the issue on RHEL 9 using Python 3.11 or 3.12, then compare fork versus spawn and the effects of savefig, close(fig), close('all'), and close(). Check the listed NumPy and Matplotlib version combinations and backend independence. Done means isolating the component or interaction responsible for the thread-allocation regression and verifying the behavior across the reported configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-visualization, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.