xarray >= 2025.8.0: Respect ulimit -n with dask.distributed
Open
Nobody has claimed this yet.
topic-dask
topic-zarr
upstream issue
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
It's hard to tell whether this is a bug in xarray or dask, but since the async update for 2025.8.0, I have been getting many OSError: [Errno 24] Too many open files errors when opening large zarr datasets. I don't have a minimal reproducible example readily available since I believe this only occurs for large files. My ulimit -n is set to 1024. Example traceback below:
Traceback
Traceback (most recent call last):
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/scheduler.py", line 4874, in update_graph
expr = deserialize(expr_ser.header, expr_ser.frames)
^^^^^^^
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 452, in deserialize
return loads(header, frames)
^^^^^^^^^^^^^^^^^
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 111, in pickle_loads
return pickle.loads(pik, buffers=buffers)
^^^^^^^^^^^^^^^^^
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/pickle.py", line 93, in loads
return pickle.loads(x, buffers=buffers)
^^^^^^^^^^^
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/zarr/storage.py", line 1835, in __setstate__
self.__init__(path=path, compression=compression, allowZip64=allowZip64, mode=mode)
^^^^^^^^^^^^^^^
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/zarr/storage.py", line 1823, in __init__
self.zf = zipfile.ZipFile(path, mode=mode, compression=compression, allowZip64=allowZip64)
^^^^^^^^^^^^^^^ File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/zipfile/__init__.py", line 1367, in __init__
self.fp = io.open(file, filemode)
^^^
OSError: [Errno 24] Too many open files: '/baril/scenario/derived/Portraits-Climatiques-v2/simulation/ESPO-G6-AHCCD_v1-patch/CMIP6/ScenarioMIP/Ouranos/MIRO
C6/ssp585/r1i1p1f1/IDM/indicators/QS-DEC/liquidprcptot/liquidprcptot_QS-DEC_ESPO-G6-AHCCD_v1-patch_CMIP6_ScenarioMIP_Ouranos_MIROC6_ssp585_r1i1p1f1_IDM_ind
icators_1950-2100.zarr.zip'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sgammon/projects/portraits/preparation/2_statistics.py", line 151, in <module>
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/core/dataset.py", line 2109, in to_netcdf
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/backends/api.py", line 2096, in to_netcdf
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/backends/common.py", line 413, in sync
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/namedarray/daskmanager.py", line 247, in store
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/dask/array/core.py", line 1227, in store
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/dask/base.py", line 681, in compute
File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/client.py", line 2417, in _gather
raise exception.with_traceback(traceback)
RuntimeError: Error during deserialization of the task graph. This frequently
occurs if the Scheduler and Client have different environments.
For more information, see
https://docs.dask.org/en/stable/deployment-considerations.html#consistent-software-environments
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the traceback paths in distributed/scheduler.py, distributed/protocol/serialize.py, zarr/storage.py, and xarray's backend and Dask manager code. Reproduce the large Zarr dataset workflow under ulimit -n 1024 and compare behavior before and after the 2025.8.0 async update. Done means identifying whether xarray or Dask causes the descriptor growth and preventing the reported OSError with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100