numpy / numpy/numpy

BUG: inf in quantile has undefined behaviour (and possibly different for -inf vs +inf)

Open
#21,091 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Describe the issue:
  1. When one or more inf or -inf are present in the argument to np.quantile (or np.nanquantile), the results often include nan, when +/-inf could be reasonably returned -- e.g. if there are 10 -infs in 100-long x, np.quantile(x,.05) should probably return -inf, not nan.

  2. np.quantile(-inf,0) =/= np.quantile(-inf,0.) (int vs float, and similarly for 1 vs 1.)

  3. The behaviour for -inf and inf is possibly different in some situations -- e.g. compare actual outputs 7 vs 9 below: the median in 7 that averages 2 and inf returns nan while the median in 9 that averages -inf and 3 returns -inf.

Likely related: #12282

Reproduce the code example:
import numpy as np
inf = np.inf
nan = np.nan
eps = 1e-9
x_pos_even = [1,2,inf,inf]
x_pos_odd  = [1,2,3,inf,inf]
x_neg_even = [-inf,-inf,3,4]
x_neg_odd  = [-inf,-inf,3,4,5]
q_even     = [0,1/3,2/3,1]
q_odd      = [0,.25,.5,.75,1]
printfun = lambda r: print(np.round(r,2))
printfun(np.quantile(x_neg_even,0))
printfun(np.quantile(x_neg_even,0.))
printfun(np.quantile(x_pos_even,q_even))
printfun(np.quantile(x_pos_odd, q_even))
printfun(np.quantile(x_neg_even,q_even))
printfun(np.quantile(x_neg_odd, q_even))
printfun(np.quantile(x_pos_even,q_odd))
printfun(np.quantile(x_pos_odd, q_odd))
printfun(np.quantile(x_neg_even,q_odd))
printfun(np.quantile(x_neg_odd, q_odd))

# expected output -- nan* = truly undefined behaviour, though could possibly be +/- inf
# -inf
# -inf
# [ 1.  , 2.  , inf , inf ]
# [ 1.  , 2.33, nan*, inf ]
# [-inf ,-inf , 3.  , 4.  ]
# [-inf , nan*, 3.67, 5.  ]
# [ 1.  , 1.75, nan*, inf , inf ]
# [ 1.  , 2.  , 3.  , inf , inf ]
# [-inf ,-inf , nan*, 3.25, 4.  ]
# [-inf ,-inf , 3.  , 4.  , 5.  ]

# actual output (spacing adjusted for readability)
# -inf
# nan
# [ 1.  , nan , nan , nan ]
# [ 1.  , 2.33, nan , nan ]
# [ nan , nan , 3.  , 4.  ]
# [ nan , nan , 3.67, 5.  ]
# [ 1.  , 1.75, nan , nan , nan ]
# [ 1.  , 2.  , nan , nan , nan ]
# [ nan , nan ,-inf , 3.25, 4.  ]
# [ nan , nan , 3.  , 4.  , 5.  ]
Error message:
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
  diff_b_a = subtract(b, a)
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4488: RuntimeWarning: invalid value encountered in multiply
  lerp_interpolation = asanyarray(add(a, diff_b_a * t, out=out))
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4489: RuntimeWarning: invalid value encountered in subtract
  subtract(b, diff_b_a * (1 - t), out=lerp_interpolation, where=t >= 0.5)
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4488: RuntimeWarning: invalid value encountered in add
  lerp_interpolation = asanyarray(add(a, diff_b_a * t, out=out))
NumPy/Python version information:

1.22.2 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the examples with np.quantile and np.nanquantile, then inspect numpy/lib/function_base.py around the interpolation code shown in the traceback. Review related issue #12282 and clarify the intended behavior for infinities, integer versus float quantiles, and warning cases. Done means the agreed behavior is consistent for positive and negative infinity and the supplied examples no longer produce unintended results.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.