numpy / numpy/numpy

BUG: Masked array default fill value can overflow

Open
#25,677 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Describe the issue:

For both signed and unsigned integers the default fill value is 99999, while for floats it is 1e20.

This is problematic for (u)int[8,16] as well as half floats, which do not contain the default fill value in their valid range.

Reproduce the code example:
>>> arr = np.ma.array([1, 2, 3], mask=[1, 0, 1], dtype=np.int8)
>>> arr.filled()
array([63,  2, 63], dtype=int8)

>>> arr = np.ma.array([1, 2, 3], mask=[1, 0, 1], dtype=np.float16)
>>> arr.filled()
/Users/goldbaum/Documents/numpy/build-install/usr/lib/python3.11/site-packages/numpy/ma/core.py:3873: RuntimeWarning: overflow encountered in cast
  np.copyto(result, fill_value, where=m)
Out[16]: array([inf,  2., inf], dtype=float16)
Error message:
N/A
Python and NumPy Versions:

Numpy 2.0 dev on python 3.11.

Runtime Environment:

N/A

Context for the issue:

I don't think this is an urgent but didn't see an issue describing this behavior so I'm filing this for future searchers.

That said, this does complicate the NEP 50 implementation because we need to have a number of workarounds so that this continues to work. I think it would be better to choose a default fill value that fits in the range of the data ([i,f]info.max?) for these types, but I have no idea what that entails for backward compatibility.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with masked-array filled() behavior in numpy/ma/core.py, around the np.copyto call shown in the warning, and reproduce the int8 and float16 examples from the issue. Determine how default fill values should behave within each dtype's range while considering the stated backward-compatibility concern. Done means the affected integer and half-float cases no longer overflow or produce the shown out-of-range results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.