Memory leak - xr.open_dataset() not releasing memory.

What happened?

Let's take this sample netcdf file.

Observe that the memory has not been cleared even after deleting the ds.

Code

import os
import psutil
import xarray as xr
from memory_profiler import profile

@profile
def main():
    path = 'ECMWF_ERA-40_subset.nc'
    print(f"Before opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")
    ds = xr.open_dataset(path)
    del ds
    print(f"After opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")

if __name__ == '__main__':
    print(f"Start: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")
    main()
    print(f"End: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")

Console logs

Start: 186.5859375 MiB
Before opening file: 187.25 MiB
After opening file: 308.09375 MiB
Filename: temp.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     6    187.2 MiB    187.2 MiB           1   @profile
     7                                         def main():
     8    187.2 MiB      0.0 MiB           1       path = 'ECMWF_ERA-40_subset.nc'
     9    187.2 MiB      0.0 MiB           1       print(f"Before opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")
    10    308.1 MiB    120.8 MiB           1       ds = xr.open_dataset(path)
    11    308.1 MiB      0.0 MiB           1       del ds
    12    308.1 MiB      0.0 MiB           1       print(f"After opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB")


End: 308.09375 MiB

I am using xarray==0.20.2and gdal==3.5.1.
Sister issue: https://github.com/ecmwf/cfgrib/issues/325#issuecomment-1363011917

What did you expect to happen?

Ideally, memory consumed by the xarray dataset should be released when the dataset is closed/deleted.

Minimal Complete Verifiable Example

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Start by reproducing the reported behavior with the provided xr.open_dataset() script and ECMWF_ERA-40_subset.nc sample file. Investigate why memory remains after deleting the dataset, and consider the issue resolved when the cause is identified and a regression check demonstrates the expected memory behavior.