[FEATURE]: Read from/write to several NetCDF4 groups with a single file open/close operation

Is your feature request related to a problem?

I know that there is a big discussion going on in #4118 about organizing hierarchies of datasets within xarray's data structures. But this issue is supposed to address only a comparably simple aspect of this.

Suppose that you have a list ds_list of xarray.Dataset objects with different dimensions etc. and you want to store them all in one NetCDF4 file by using the group feature introduced in NetCDF4. The group name of each dataset is stored in ds_names. Obviously, you can do something like this:

for name, ds in zip(ds_names, ds_list):
    ds.to_netcdf(path, group=name)

However, this is really slow when you have many (hundreds or thousands of) small datasets because the file is opened and closed in every iteration.

Describe the solution you'd like

I would like to have a function xr.to_netcdf that writes a list (or a dictionary) of datasets to a single NetCDF4 file with a single open/close operation. Ideally there should also be a way to read many datasets at once from a single NetCDF4 file using xr.open_dataset.

Describe alternatives you've considered

Currently, I'm using the following read/write functions to achieve the same:

import pathlib
from xarray.backends import NetCDF4DataStore
from xarray.backends.api import dump_to_store
from xarray.backends.common import ArrayWriter
from xarray.backends.store import StoreBackendEntrypoint

def _xr_to_netcdf_multi(path, ds_dict, encoding=None):
    """Write multiple xarray Datasets to separate groups in a single NetCDF4 file

    Parameters
    ----------
    path : str or Path
        Path of the target NetCDF file.
    ds_dict : dict whose keys are group names and values are xr.Dataset
        Each xr.Dataset in the dict is stored in the group identified by its key in the dict.
        Note that an empty string ("") is a valid group name and refers to the root group.
    encoding : dict whose keys are group names and values are dict, optional
        For each dataset/group, one dict that is compliant with the format of the `encoding`
        keyword parameter in `xr.Dataset.to_netcdf`. Default: None
    """
    path = str(pathlib.Path(path).expanduser().absolute())
    store = NetCDF4DataStore.open(path, "w", "NETCDF4", None)
    try:
        writer = ArrayWriter()
        for group, dataset in ds_dict.items():
            store._group = group
            unlimited_dims = dataset.encoding.get("unlimited_dims", None)
            encoding = None if encoding is None or group not in encoding else encoding[group]
            dump_to_store(dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims)
    finally:
        store.close()

def _xr_open_dataset_multi(path, prefix=""):
    """Read multiple xarray Datasets from groups contained in a single NetCDF4 file

    Warning: The data is loaded into memory!

    Parameters
    ----------
    path : str or Path
        Path of the NetCDF file to read.
    prefix : str, optional
        If given, only read groups whose name starts with this prefix. Default: ""

    Returns
    -------
    ds_dict : dict whose keys are group names and values are xr.Dataset
        Each xr.Dataset in the dict is taken from the group identified by its key in the dict.
        Note that an empty string ("") is a valid group name and refers to the root group.
    """
    path = str(pathlib.Path(path).expanduser().absolute())
    store = NetCDF4DataStore.open(path, "r", "NETCDF4", None)
    ds_dict = {}
    try:
        groups = [g for g in _xr_nc4_groups_from_store(store) if g.startswith(prefix)]
        store_entrypoint = StoreBackendEntrypoint()
        for group in groups:
            store._group = group
            ds = store_entrypoint.open_dataset(store)
            ds.load()
            ds_dict[group] = ds
    finally:
        store.close()
    return ds_dict

def _xr_nc4_groups_from_store(store):
    """List all groups contained in the given NetCDF4 data store

    Parameters
    ----------
    store : xarray.backend.NetCDF4DataStore

    Returns
    -------
    list of str
    """
    def iter_groups(ds, prefix=""):
        groups = [""]
        for group_name, group_ds in ds.groups.items():
            groups.extend([f"{prefix}{group_name}{subgroup}"
                           for subgroup in iter_groups(group_ds, prefix="/")])
        return groups
    with store._manager.acquire_context(False) as root:
        return iter_groups(root)

Additional context

No response

Start with Dataset.to_netcdf and open_dataset, then inspect the NetCDF4DataStore, dump_to_store, ArrayWriter, and StoreBackendEntrypoint APIs shown in the issue. Review discussion #4118 for how this feature should relate to dataset hierarchies. Done means multiple datasets can be read or written across NetCDF4 groups with one file open/close operation.