matplotlib / matplotlib/matplotlib
[Bug]: Matplotlib savefig and close break multiprocessing and numpy thread allocation
Nadie ha tomado este issue todavía.
- Lenguaje dominante
- Python
- Estrellas
- 23.2k
- Forks
- 8.5k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 66
Descripción
### Bug summary
I tried reporting this as a python bug but they said they don't really deal with bugs depending on third party packages, so I am trying here. I recently updated my python environment and discovered a weird bug that occurs between matplotlib, numpy, and multiprocessing.
Code to demonstrate the bug below. In the code I am running a multiprocessing pool, with a function that multiplies small matrices a large number of times. If run in a fresh environment (from commmand line or in new ipython session) the loops runs once quickly, utilizing all cpu threads. Then a plot is generated, saved and closed, and the loop runs again, but this time only 2 threads are utilized at 100%, and the execution time is much slower.
Time is used only to demonstrate that the code runs much slower when the bug is occurring, on my main machine (48 cores/96 threads) it runs about 30 times slower when the bug occurs. I tested on another machine I have access to with 8 cores/16 threads, the bugged version runs about 10 times slower.
The try/except is used just so the pool is closed properly if you keyboard interrupt.
### Code for reproduction
```Python
import numpy as np
import multiprocessing as mp
from matplotlib import pyplot as plt
import time
def test_function(var):
test_mat=np.zeros((3,3))
for i in range(10**6):
np.matmul(test_mat,test_mat)
return()
def main():
print('starting')
run_multiplier=2
num_threads=mp.cpu_count()#//2
pool=mp.Pool(num_threads)
try:
completed=0
tic=time.time()
for item in pool.imap(test_function,range(num_threads*run_multiplier)):
completed+=1
toc=time.time()-tic
print('Elapsed time without bug %f seconds'%toc)
pool.close()
pool.join()
except:
pool.terminate()
pool.join()
test_array=np.arange(10)
fig=plt.figure()
plt.plot(test_array,test_array,'.')
plt.savefig('test_fig.png')
#plt.close('all')
plt.close(fig)
# plt.close()
pool=mp.Pool(num_threads)
try:
completed=0
tic=time.time()
for item in pool.imap(test_function,range(num_threads*run_multiplier)):
completed+=1
toc=time.time()-tic
print('Elapsed time with bug %f seconds'%toc)
pool.close()
pool.join()
except:
pool.terminate()
pool.join()
if __name__=='__main__':
main()
```
### Actual outcome
When bug occurs 2 threads are run at 100%, the rest have low utilization.

### Expected outcome
When bug does not occur all threads are near 100% utilization until the loop completes.

### Additional information
The bug occurs when either plt.savefig or plt.close are called, although I tested a few different versions of matplotlib, and the behavior changes slightly. With python 12 and matplotlib 3.10.0 the bug occurs with plt.close(fig) and plt.close('all'). With matplotlib 3.9.2 and 3.8.0 the bug does not occur with plt.close('all') but still occurs with plt.close(fig). The bug does not occur with a plain plt.close() in my testing.
This bug also occurs on python 11 with matplotlib 3.8.0, I didn't test other versions. The bug does NOT occur on python 10 for matplotlib 3.7 or 3.10. The bug also does not occur for python 3.8, matplotlib version 3.3.
The bug only occurs when the multiprocessing pool is generated using fork, not spawn (fork is only default on linux, and not available on windows). This suggests matplotlib possibly has a memory leak that is getting copied in the fork, although I also think there must be improper memory access in either multiprocessing or numpy to be reading the memory leaked by matplotlib. Alternatively some kind of threading/multiprocessing has been introduced for matplotlib in newer python version (>=3.11), but it is clashing with the thread allocation in numpy and multiprocessing.
If this is tested in an ipython environment, you will need to restart the kernel. The bug occurs if any figure has ever been saved or closed (depending on the close method) within the ipython session, so if you don't restart the session then the bugged version will just be running twice.
The bug appears to be independent of the backend chosen for matplotlib, I tried qt, tkinter, ipympl, and Agg.
Both computers I have experienced this bug on are running RHEL 9. When the code is run on WSL running Ubuntu the bug does not occur. Apparently this is somehow distribution dependent.
### Operating system
Red Hat Enterprise Linux 9
### Matplotlib Version
3.8, 3.9, and 3.10
### Matplotlib Backend
any
### Python version
3.11 and 3.12
### Jupyter version
n/a
### Installation
conda
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Línea de trabajo
Comienza con el script de reproducción del issue en RHEL 9 usando Python 3.11 o 3.12; después, compara fork frente a spawn y los efectos de savefig, close(fig), close('all') y close(). Comprueba las combinaciones de versiones de NumPy y Matplotlib indicadas y la independencia del backend. Se considera terminado cuando se haya aislado el componente o la interacción responsable de la regresión en la asignación de hilos y se haya verificado el comportamiento en las configuraciones indicadas.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- numpy, python
- Área
- data-visualization, performance
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100