N-dimensional boolean indexing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Currently, the docs state that boolean indexing is only possible with 1-dimensional arrays:
http://xarray.pydata.org/en/stable/indexing.html
However, I often have the case where I'd like to convert a subset of an xarray to a dataframe.
Usually, I would call e.g.:
data = xrds.stack(observations=["dim1", "dim2", "dim3"])
data = data.isel(~ data.missing)
df = data.to_dataframe()
However, this approach is incredibly slow and memory-demanding, since it creates a MultiIndex of every possible coordinate in the array.
Describe the solution you'd like
A better approach would be to directly allow index selection with the boolean array:
data = xrds.isel(~ xrds.missing, dim="observations")
df = data.to_dataframe()
This way, it is possible to
- Identify the resulting coordinates with
np.argwhere() - Directly use the underlying array for fancy indexing:
variable.data[mask]
Additional context
I created a proof-of-concept that works for my projects:
https://gist.github.com/Hoeze/c746ea1e5fef40d99997f765c48d3c0d
Some important lines are those:
def core_dim_locs_from_cond(cond, new_dim_name, core_dims=None) -> List[Tuple[str, xr.DataArray]]:
[...]
core_dim_locs = np.argwhere(cond.data)
if isinstance(core_dim_locs, dask.array.core.Array):
core_dim_locs = core_dim_locs.persist().compute_chunk_sizes()
def subset_variable(variable, core_dim_locs, new_dim_name, mask=None):
[...]
subset = dask.array.asanyarray(variable.data)[mask]
# force-set chunk size from known chunks
chunk_sizes = core_dim_locs[0][1].chunks[0]
subset._chunks = (chunk_sizes, *subset._chunks[1:])
As a result, I would expect something like this:

Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the indexing documentation and the isel/to_dataframe examples in the issue, then review the proof-of-concept, especially core_dim_locs_from_cond and subset_variable. Compare the proposed np.argwhere and underlying-array indexing behavior, including the dask case. Done means n-dimensional boolean selection works without constructing every possible coordinate and preserves usable coordinates for to_dataframe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100