pydata / pydata/xarray

Allow skipna in .dot()

Open
#4,482 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic-performance
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem? Please describe.
Right now there's no efficient way to do a dot product that skips over nan elements.

Describe the solution you'd like
I want to be able to treat the summation in dot as a nansum, controlled by a skipna option. Either this can be implemented directly, or an additional ufunc can be added: xarray.unfuncs.nan_to_num that can be called on the inputs to dot. Unfortunately using numpy's nan_to_num will initiate eager execution.

Describe alternatives you've considered
It's possible to implement this by hand, but it ends up being extremely inefficient in one of my use-cases:

  • (x*y).sum('dot_prod_dim', skipna=True) takes 30 seconds
  • x.dot(y) takes 1 second

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the .dot entry point and the existing tests for dot products and skipna reductions. Check how the current implementation avoids eager execution, then define tests covering skipna behavior with NaN values and the existing behavior without it. Done means .dot() accepts the requested option and handles NaNs efficiently without eager execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.