BUG: inf in quantile has undefined behaviour (and possibly different for -inf vs +inf)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Describe the issue:
-
When one or more
infor-infare present in the argument tonp.quantile(ornp.nanquantile), the results often includenan, when+/-infcould be reasonably returned -- e.g. if there are 10-infs in 100-longx,np.quantile(x,.05)should probably return-inf, notnan. -
np.quantile(-inf,0)=/=np.quantile(-inf,0.)(int vs float, and similarly for1vs1.) -
The behaviour for
-infandinfis possibly different in some situations -- e.g. compare actual outputs 7 vs 9 below: the median in 7 that averages 2 andinfreturnsnanwhile the median in 9 that averages-infand 3 returns-inf.
Likely related: #12282
Reproduce the code example:
import numpy as np
inf = np.inf
nan = np.nan
eps = 1e-9
x_pos_even = [1,2,inf,inf]
x_pos_odd = [1,2,3,inf,inf]
x_neg_even = [-inf,-inf,3,4]
x_neg_odd = [-inf,-inf,3,4,5]
q_even = [0,1/3,2/3,1]
q_odd = [0,.25,.5,.75,1]
printfun = lambda r: print(np.round(r,2))
printfun(np.quantile(x_neg_even,0))
printfun(np.quantile(x_neg_even,0.))
printfun(np.quantile(x_pos_even,q_even))
printfun(np.quantile(x_pos_odd, q_even))
printfun(np.quantile(x_neg_even,q_even))
printfun(np.quantile(x_neg_odd, q_even))
printfun(np.quantile(x_pos_even,q_odd))
printfun(np.quantile(x_pos_odd, q_odd))
printfun(np.quantile(x_neg_even,q_odd))
printfun(np.quantile(x_neg_odd, q_odd))
# expected output -- nan* = truly undefined behaviour, though could possibly be +/- inf
# -inf
# -inf
# [ 1. , 2. , inf , inf ]
# [ 1. , 2.33, nan*, inf ]
# [-inf ,-inf , 3. , 4. ]
# [-inf , nan*, 3.67, 5. ]
# [ 1. , 1.75, nan*, inf , inf ]
# [ 1. , 2. , 3. , inf , inf ]
# [-inf ,-inf , nan*, 3.25, 4. ]
# [-inf ,-inf , 3. , 4. , 5. ]
# actual output (spacing adjusted for readability)
# -inf
# nan
# [ 1. , nan , nan , nan ]
# [ 1. , 2.33, nan , nan ]
# [ nan , nan , 3. , 4. ]
# [ nan , nan , 3.67, 5. ]
# [ 1. , 1.75, nan , nan , nan ]
# [ 1. , 2. , nan , nan , nan ]
# [ nan , nan ,-inf , 3.25, 4. ]
# [ nan , nan , 3. , 4. , 5. ]
Error message:
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
diff_b_a = subtract(b, a)
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4488: RuntimeWarning: invalid value encountered in multiply
lerp_interpolation = asanyarray(add(a, diff_b_a * t, out=out))
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4489: RuntimeWarning: invalid value encountered in subtract
subtract(b, diff_b_a * (1 - t), out=lerp_interpolation, where=t >= 0.5)
/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:4488: RuntimeWarning: invalid value encountered in add
lerp_interpolation = asanyarray(add(a, diff_b_a * t, out=out))
NumPy/Python version information:
1.22.2 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the examples with np.quantile and np.nanquantile, then inspect numpy/lib/function_base.py around the interpolation code shown in the traceback. Review related issue #12282 and clarify the intended behavior for infinities, integer versus float quantiles, and warning cases. Done means the agreed behavior is consistent for positive and negative infinity and the supplied examples no longer produce unintended results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100