Add `fill` to `.sel` for missing values.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
It would be ideal if the .sel accessors provided a fill mode that nan-fills, rather than errors,
when a value isn't present in the target index.
This often occurs when a Dataset or DataArray is indexed by an arbitrary categorial value,
and we want to index against that value by an arbitrarily-dimensioned coordinate from another array.
If the coordinate is missing in the target index I want a (conceptual) outer join, rather than erroring.
It would be useful to add a fill_value parameter that's used like fillna so that these missing indices can be filled with user-provided value, however this might be dangerous because .sel already accepts kwargs.
Constraining this to when it's explicitly requested via mode="fill" or the other fill modes would make this safer.
This is related to https://github.com/pydata/xarray/issues/4995
Describe the solution you'd like
Adding a fill mode to sel, which performs nan-filling, rather than raising a KeyError, when values are missing from the index.
If mode="fill" is specified, allow an additional fill_value kwarg which specifies the fill value with the same semantics as fillna.
Potentially allow this filling logic for other modes that would currently raise a KeyError on failed fill operations.
This is a restricted, single-dimension, inefficient workaround that implements this logic for a single indexing dimension:
def sel_fill(
left: XA,
dim: str,
right_coord: xa.DataArray,
fill_value: Any | dict[str, Any] = xa.core.dtypes.NA, # type: ignore
) -> XA:
""".sel but return nan/_fill for missing values.
This is a "right join" when indexing a dim with a coord array.
For example, for "data" with a dim matching a coord of "samples":
`data.sel(name=samples.sample_name)` is a strict join...
...if a value in sample_name isn't present in data.name it's an error.
However, you may want to select-and-fill-missing-values...
...if a value in present in data.name, then select data.
...otherwise return a nan or filled value.
This uses the same filling logic as xa.align,
provide a dictionary of names to fill with specific values.
"""
assert dim in left.indexes
index: pd.Index = left.indexes[dim]
coord_index = pd.Index(right_coord.values.ravel())
missing_values = coord_index.difference(index)
if missing_values.empty:
left_data = left
else:
left_data = xa.concat(
[
left,
left.reindex({dim: missing_values}, fill_value=fill_value),
],
dim=dim,
)
return left_data.sel({dim: right_coord})
Describe alternatives you've considered
.sel- Raises a KeyError if an coordinate isn't present in the index. Can perform fill for some numeric indexes, but doesn't have a fill operation for discrete indexes..reindex- Functions when the "query" or "right" coordinate has the same dims as the "value" or "left" data, however we can't reindex against coordinates with different dimensionality that the value coordinate.
Additional context
Semi-MVP repro:
import xarray as xa
import numpy as np
import pandas as pd
from typing import Any
def sel_fill(
left: xa.DataArray,
dim: str,
right_coord: xa.DataArray,
fill_value: Any | dict[str, Any] = xa.core.dtypes.NA, # type: ignore
) -> xa.DataArray:
""".sel but return nan/_fill for missing values.
This is a "right join" when indexing a dim with a coord array.
For example, for "data" with a dim matching a coord of "samples":
`data.sel(name=samples.sample_name)` is a strict join...
...if a value in sample_name isn't present in data.name it's an error.
However, you may want to select-and-fill-missing-values...
...if a value in present in data.name, then select data.
...otherwise return a nan or filled value.
This uses the same filling logic as xa.align,
provide a dictionary of names to fill with specific values.
"""
assert dim in left.indexes
index: pd.Index = left.indexes[dim]
coord_index = pd.Index(right_coord.values.ravel())
missing_values = coord_index.difference(index)
if missing_values.empty:
left_data = left
else:
left_data = xa.concat(
[
left,
left.reindex({dim: missing_values}, fill_value=fill_value),
],
dim=dim,
)
return left_data.sel({dim: right_coord})
dat = xa.DataArray(np.arange(4), dims="letter").assign_coords(letter=list("abcd"))
query = xa.DataArray([["a", "b", "c"], ["d", "e", "f"]])
# KeyError: "not all values found in index 'letter'"
dat.sel(letter=query)
# ValueError: Indexer has dimensions ('dim_0', 'dim_1') that are different from that to be indexed along 'letter'
dat.reindex(dict(letter=query))
# Just-right
sel_fill(dat, "letter", query, fill_value=1663)
<xarray.DataArray (dim_0: 2, dim_1: 3)>
array([[ 0, 1, 2],
[ 3, 1663, 1663]])
Coordinates:
letter (dim_0, dim_1) object 'a' 'b' 'c' 'd' 'e' 'f'
Dimensions without coordinates: dim_0, dim_1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the semi-MVP example and read the existing .sel behavior described in the issue, including how missing index values and multidimensional indexers are handled. Define the supported mode="fill" and fill_value semantics, including whether other modes are affected. Done means missing categorical selections return the requested fill values without raising a KeyError, while existing selections retain their current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100