open_mfdataset memory leak, very simple case. v0.12
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
MCVE Code Sample
import glob
import xarray as xr
import numpy as np
from memory_profiler import profile
def CreateTestFiles():
# create a bunch of files
xlen = int(1e2)
ylen = int(1e2)
xdim = np.arange(xlen)
ydim = np.arange(ylen)
nfiles = 100
for i in range(nfiles):
data = np.random.rand(xlen, ylen, 1)
datafile = xr.DataArray(data, coords=[xdim, ydim, i], dims=['x', 'y', 'time'])
datafile.to_netcdf('testfiles/datafile_{}.nc'.format(i))
@profile
def ReadFiles():
xr.open_mfdataset(glob.glob('testfiles/*'), concat_dim='time')
if __name__ == '__main__':
# write out files for testing
CreateTestFiles()
# loop thru file read step
for i in range(100):
ReadFiles()
~
~
usage:
mprof run simplest_case.py
mprof plot
(mprof is a python memory profiling library)
Problem Description
dask version 1.1.4
xarray version 0.12
python 3.7.3
There appears to be a persistent memory leak in open_mfdataset. I'm creating a model calibration script that runs for ~1000 iterations, opening and closing the same set of files (dimensions are the same, but the data is different) with each iteration. I eventually run out of memory because of the leak. This simple case captures the same behavior. Closing the files with .close() does not fix the problem.
Is there a work around for this? I've perused some of the issues but cannot tell if this has been resolved.

Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.17.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.2
xarray: 0.12.0
pandas: 0.24.2
numpy: 1.16.2
scipy: 1.2.1
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: None
Nio: 1.5.5
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudonetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 1.1.4
distributed: 1.26.0
matplotlib: 3.0.2
cartopy: 0.17.0
seaborn: None
setuptools: 41.0.1
pip: 19.1.1
conda: None
pytest: None
IPython: 7.3.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The entry point is xr.open_mfdataset in the supplied simplest_case.py MCVE; start by running it under mprof with the listed versions and inspecting repeated calls. Compare memory behavior with the described close() attempt. Done means repeated reads no longer show persistent growth, with the behavior covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100