BUG: Masked array default fill value can overflow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Describe the issue:
For both signed and unsigned integers the default fill value is 99999, while for floats it is 1e20.
This is problematic for (u)int[8,16] as well as half floats, which do not contain the default fill value in their valid range.
Reproduce the code example:
>>> arr = np.ma.array([1, 2, 3], mask=[1, 0, 1], dtype=np.int8)
>>> arr.filled()
array([63, 2, 63], dtype=int8)
>>> arr = np.ma.array([1, 2, 3], mask=[1, 0, 1], dtype=np.float16)
>>> arr.filled()
/Users/goldbaum/Documents/numpy/build-install/usr/lib/python3.11/site-packages/numpy/ma/core.py:3873: RuntimeWarning: overflow encountered in cast
np.copyto(result, fill_value, where=m)
Out[16]: array([inf, 2., inf], dtype=float16)
Error message:
N/A
Python and NumPy Versions:
Numpy 2.0 dev on python 3.11.
Runtime Environment:
N/A
Context for the issue:
I don't think this is an urgent but didn't see an issue describing this behavior so I'm filing this for future searchers.
That said, this does complicate the NEP 50 implementation because we need to have a number of workarounds so that this continues to work. I think it would be better to choose a default fill value that fits in the range of the data ([i,f]info.max?) for these types, but I have no idea what that entails for backward compatibility.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with masked-array filled() behavior in numpy/ma/core.py, around the np.copyto call shown in the warning, and reproduce the int8 and float16 examples from the issue. Determine how default fill values should behave within each dtype's range while considering the stated backward-compatibility concern. Done means the affected integer and half-float cases no longer overflow or produce the shown out-of-range results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100