pydata / pydata/xarray

Add `fill` to `.sel` for missing values.

Open
#8,881 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

It would be ideal if the .sel accessors provided a fill mode that nan-fills, rather than errors,
when a value isn't present in the target index.

This often occurs when a Dataset or DataArray is indexed by an arbitrary categorial value,
and we want to index against that value by an arbitrarily-dimensioned coordinate from another array.
If the coordinate is missing in the target index I want a (conceptual) outer join, rather than erroring.

It would be useful to add a fill_value parameter that's used like fillna so that these missing indices can be filled with user-provided value, however this might be dangerous because .sel already accepts kwargs.
Constraining this to when it's explicitly requested via mode="fill" or the other fill modes would make this safer.

This is related to https://github.com/pydata/xarray/issues/4995

Describe the solution you'd like

Adding a fill mode to sel, which performs nan-filling, rather than raising a KeyError, when values are missing from the index.
If mode="fill" is specified, allow an additional fill_value kwarg which specifies the fill value with the same semantics as fillna.

Potentially allow this filling logic for other modes that would currently raise a KeyError on failed fill operations.

This is a restricted, single-dimension, inefficient workaround that implements this logic for a single indexing dimension:

def sel_fill(
    left: XA,
    dim: str,
    right_coord: xa.DataArray,
    fill_value: Any | dict[str, Any] = xa.core.dtypes.NA,  # type: ignore
) -> XA:
    """.sel but return nan/_fill for missing values.

    This is a "right join" when indexing a dim with a coord array.
    For example, for "data" with a dim matching a coord of "samples":

    `data.sel(name=samples.sample_name)` is a strict join...
    ...if a value in sample_name isn't present in data.name it's an error.
    However, you may want to select-and-fill-missing-values...
    ...if a value in present in data.name, then select data.
    ...otherwise return a nan or filled value.

    This uses the same filling logic as xa.align,
    provide a dictionary of names to fill with specific values.

    """
    assert dim in left.indexes
    index: pd.Index = left.indexes[dim]

    coord_index = pd.Index(right_coord.values.ravel())

    missing_values = coord_index.difference(index)

    if missing_values.empty:
        left_data = left
    else:
        left_data = xa.concat(
            [
                left,
                left.reindex({dim: missing_values}, fill_value=fill_value),
            ],
            dim=dim,
        )

    return left_data.sel({dim: right_coord})
Describe alternatives you've considered
  • .sel - Raises a KeyError if an coordinate isn't present in the index. Can perform fill for some numeric indexes, but doesn't have a fill operation for discrete indexes.
  • .reindex - Functions when the "query" or "right" coordinate has the same dims as the "value" or "left" data, however we can't reindex against coordinates with different dimensionality that the value coordinate.
Additional context

Semi-MVP repro:

import xarray as xa
import numpy as np
import pandas as pd
from typing import Any

def sel_fill(
    left: xa.DataArray,
    dim: str,
    right_coord: xa.DataArray,
    fill_value: Any | dict[str, Any] = xa.core.dtypes.NA,  # type: ignore
) -> xa.DataArray:
    """.sel but return nan/_fill for missing values.

    This is a "right join" when indexing a dim with a coord array.
    For example, for "data" with a dim matching a coord of "samples":

    `data.sel(name=samples.sample_name)` is a strict join...
    ...if a value in sample_name isn't present in data.name it's an error.
    However, you may want to select-and-fill-missing-values...
    ...if a value in present in data.name, then select data.
    ...otherwise return a nan or filled value.

    This uses the same filling logic as xa.align,
    provide a dictionary of names to fill with specific values.

    """
    assert dim in left.indexes
    index: pd.Index = left.indexes[dim]

    coord_index = pd.Index(right_coord.values.ravel())

    missing_values = coord_index.difference(index)

    if missing_values.empty:
        left_data = left
    else:
        left_data = xa.concat(
            [
                left,
                left.reindex({dim: missing_values}, fill_value=fill_value),
            ],
            dim=dim,
        )

    return left_data.sel({dim: right_coord})

dat = xa.DataArray(np.arange(4), dims="letter").assign_coords(letter=list("abcd"))
query = xa.DataArray([["a", "b", "c"], ["d", "e", "f"]])

# KeyError: "not all values found in index 'letter'"
dat.sel(letter=query)

# ValueError: Indexer has dimensions ('dim_0', 'dim_1') that are different from that to be indexed along 'letter'
dat.reindex(dict(letter=query))

# Just-right
sel_fill(dat, "letter", query, fill_value=1663)
<xarray.DataArray (dim_0: 2, dim_1: 3)>
array([[   0,    1,    2],
       [   3, 1663, 1663]])
Coordinates:
    letter   (dim_0, dim_1) object 'a' 'b' 'c' 'd' 'e' 'f'
Dimensions without coordinates: dim_0, dim_1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the semi-MVP example and read the existing .sel behavior described in the issue, including how missing index values and multidimensional indexers are handled. Define the supported mode="fill" and fill_value semantics, including whether other modes are affected. Done means missing categorical selections return the requested fill values without raising a KeyError, while existing selections retain their current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.