numpy / numpy/numpy

Incorrect histogramdd behavior with masked arrays

Open
#5,363 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug component: numpy.lib
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Summing over the histogram should always give the number of non-masked records when no weighting is used. As shown below, the sum is reduced, but not by the correct amount. Calling the "compressed()" method and reshaping before running the histogram produces correct results.

The following uses numpy 1.8.1.

 x = ma.arange(300).reshape(100,3)

In [5]: print x[:,0].count()
100

In [6]: H,e = np.histogramdd(x)

In [7]: H.sum()
Out[7]: 100.0

In [8]: for i in range(10) : 
   ...:     x[i*i,:]=ma.masked
   ...:     

In [9]: print x[:,0].count()
90

In [10]: H,e = np.histogramdd(x)

In [11]: H.sum()
Out[11]: 98.0

x = x.compressed().reshape(90,3)

H,e = np.histogramdd(x)

H.sum()
Out[14]: 90.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the masked-array example with numpy.histogramdd and compare its result with x.compressed().reshape(90,3). Check how masked values are handled during histogramdd and review the existing histogramdd tests, if available. Done means an unweighted histogram sums to the number of non-masked records.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.