`open_mfdataset` hangs indefinitely with h5netcdf and dask since commit ea9f02 -> v 2025.9.1

What happened?

Description
Since https://github.com/pydata/xarray/commit/ea9f02bbe6d3b02fbb56600710b2792795e0e4a5 and #10571, the following snippet hangs indefinitely — the files are opened successfully, but xr.open_mfdataset seems to hang and never completes. I end up having to kill the process. Reverting to the commit just before this change works fine.

open_mfdataset_params = {
    "engine": engine,
    "parallel": True,
    "preprocess": partial_preprocess,
    "data_vars": "all",
    "concat_characters": True,
    "mask_and_scale": True,
    "decode_cf": True,
    "decode_times": self.time_coder,
    "decode_coords": True,
    "compat": "override",
    "coords": "minimal",
    "drop_variables": drop_vars_list,
}

ds = xr.open_mfdataset(batch_files, **open_mfdataset_params)

Context

• Engine: h5netcdf
• Scheduler: Dask running on a Coiled cluster
• Input: batch_files is a fileset array of NetCDF objects stored in S3 (fileset = [s3_fs.open(file) for file in s3_paths]
• preprocess runs successfully on all files, but open_mfdataset itself gets stuck

What I’ve checked

• I’m aware of https://github.com/pydata/xarray/issues/10712, and like @rabernat, that change broke my code. I actually caught it in my own integration tests, whereas the xarray unit tests didn’t expose it at the time. More generally, I think some of these regressions could be avoided if there were higher-level integration tests in addition to the existing unit tests. That particular issue was fixed about two weeks ago and resolved the pickle error I was seeing. However, I’m still encountering the hang I describe here.

• Rolling back to the commit before https://github.com/pydata/xarray/commit/ea9f02bbe6d3b02fbb56600710b2792795e0e4a5 avoids the problem. In my case, 938e18680cbd9649f375d1544af8e5a16c352210 is the latest working commit

What did you expect to happen?

xr.open_mfdataset should complete successfully, as it does on earlier commits.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import xarray as xr
xr.show_versions()
# your reproducer code ...

Steps to reproduce

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

No response

Environment

No source file or test is named. Start at xr.open_mfdataset with engine='h5netcdf' and parallel=True, using the listed parameters and S3-backed fileset context, then compare behavior before and after commit ea9f02. Done means the call completes instead of hanging indefinitely.