pydata / pydata/xarray

Allow sel's method and tolerance to vary per-dimension

Open
#4,714 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-indexing
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Imagine some data like this:

sensor_data = xr.DataArray(np.arange(6).reshape((3, 2)), coords=[
    ('time', [0, 2, 3]),
    ('sensor', ['A', 'C']),
])

Let's say we now want to sample these sensors at some arbitrary points. We can use vectorized indexing do this:

sensor_data.sel({
    'sensor': xr.DataArray(['A', 'A', 'A', 'B', 'C'], dims=['sample']),
    'time': xr.DataArray([0, 1, 2, 0, 0], dims=['sample'])
})

This fails because we are sampling one of our sensors at time 1, where we don't have any observations. We can add method='ffill' to fix this:

sensor_data.sel({
    'sensor': xr.DataArray(['A', 'A', 'A', 'B', 'C'], dims=['sample']),
    'time': xr.DataArray([0, 1, 2, 0, 0], dims=['sample'])
}, method='ffill')
# array([0, 0, 2, 0, 1])

The problem is that the bogus sensor "B" is now getting a value ffilled from sensor "A"'s time 0 observation, which doesn't make a lot of sense because sensor names are arbitary. What we really want to do is apply the ffill only down the "time" array, so the sel call sitll fails if a sensor name is unknown but we can still benefit from ffilling in places where it makes sense.

So, it would be nice if we could supply a per-dimension method (or tolerance) like this:

sensor_data.sel({
    'sensor': xr.DataArray(['A', 'A', 'A', 'B', 'C'], dims=['sample']),
    'time': xr.DataArray([0, 1, 2, 0, 0], dims=['sample'])
}, method={'time': 'ffill'})

From an implementation point of view, this looks like an easy addition in indexing.remap_label_indexers: https://github.com/pydata/xarray/blob/235b2e5bcec253ca6a85762323121d28c3b06038/xarray/core/indexing.py#L243

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xarray/core/indexing.py at indexing.remap_label_indexers, the implementation point identified in the issue, and trace how sel handles method and tolerance. Done means a per-dimension method such as {'time': 'ffill'} applies only to that dimension while an unknown sensor still causes the selection to fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.