Fancy isel on LazilyIndexedArray allocates memory of about the size of backing dataset

What happened?

I ran into a large memory allocation when doing an isel on a zarr dataset with dask turned off (ie opened with open_dataset("mydataset.zarr", chunks=None)), which caused the Linux OOM killer to kill my process.

I used some memory profilers (memray + scalene) to dig in a little; it appears that a fancy isel on a LazilyIndexedArray ultimately causes indexing._combine_indexers to run np.brodcast_arrays(…) on a set of tuples which causes the broadcast to return an numpy array of the same shape as the underlying dataset. See MCVE for an example isel which I think tries to allocate exabytes of ram. (Not loading the data; when doing the lazy isel.)

The VectorizedIndexer created at the end of _combine_indexers also ends up being made of ndarrays of the size of the data-to-be-selected which could also be large.

What did you expect to happen?

I was hoping that I could index into a large dataset cheaply before loading it (eg by tracking ranges to be selected as slices). The original example and the one below both try to select a relatively small amount of the total data.

Minimal Complete Verifiable Example

import xarray as xr
from xarray.core import indexing
import numpy as np

def test_dataset_indexing_memory() -> None:
    # Mimics open_dataset with a large backing array and no dask.
    big_backing_array = indexing.LazilyIndexedArray(
        np.broadcast_to([1], (1_000_000, 1_000_000, 1_000_000))
    )
    data_array = xr.DataArray(
        big_backing_array,
        coords={
            "t": np.arange(1_000_000),
            "y": np.arange(1_000_000),
            "z": np.arange(1_000_000),
        },
    )
    print("running isel")
    # I think any fancy isel will do
    data_array.isel(t=xr.Variable(("new", "t"), [[0]]))

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

Happy to share memory profiles or other debugging info if that would be helpful. Thank you for taking a look!

Environment

Start by running the minimal example and tracing xarray.core.indexing._combine_indexers and LazilyIndexedArray, which the report identifies as the relevant entry points. The work is done when the fancy isel selection remains lazy without allocating arrays proportional to the backing dataset, while preserving the expected selected result.