BUG: np.ma.corrcoef returns incorrect data
Open
Nobody has claimed this yet.
00 - Bug
component: numpy.ma
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Describe the issue:
I am using np.ma.corrcoef on masked 2d data with varying gaps.
- np.ma.corrcoef returns values well above 1 and below -1.
- additionally returned data values within +-1 are sometimes wrong.
When using corrcoef to obtain correlation between the single vectors:
- np.corrcoef produces correct values when reducing vectors to overlapping data (unmasked), regardless of using np.corrcoef x and y array inputs
- np.ma.corrcoef produces correct values when using masked vectors for x and y
- np.ma.corrcoef produces incorrect values when using a masked array for x
Thanks for any help. Please retag if this is not a Bug but an user error.
(Sorry for the large sample data, I tried to cut it down best I can to reproduce the numbers)
Reproduce the code example:
import numpy as np
data = np.array([[np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
0.05416667461395264, 0.3083333174387614, 1.1375001271565754, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, 0.12083332737286885,
0.3583333492279053, 0.23749999205271402, 0.4541666905085246, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan],
[np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.19583332538604736,
0.22083336114883423, 0.5874999761581421, 0.20833333333333334,
1.0708333651224773, 1.6249998410542805, 0.23749999205271402,
0.033333333830038704, 0.0, 0.0, 0.14999999602635702, 0.0,
0.07916667064030965, 0.0, 0.0, 0.0, 0.004166666728754838,
0.22916666666666666, 0.5416666666666666, 0.5458333094914755,
0.833333412806193, 0.5124999682108561, 1.3375000953674316,
1.9749999046325684, 0.07500000298023224, 0.0, 0.0,
0.012500000496705374, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.004166666728754838, 0.0, 0.0, 0.23333332935969034,
0.5250000158945719, 0.0, 0.016666666915019352],
[0.12916666269302368, 0.0, 0.0, 0.037499999006589256,
0.033333333830038704, 0.0, 0.033333333830038704,
0.14166666070620218, 0.0, 0.12916666269302368,
0.12916667262713113, 0.0, 0.0, 0.0, 0.0, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.029166666169961292, 0.9416666030883789,
0.10416667660077412, 0.1958333452542623, 0.27916667858759564,
0.7833333015441895, 0.020833333333333332, 0.050000001986821495,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.02916666865348816,
0.22916666666666666, 0.24999998013178507, 0.125,
0.20833333333333334, 0.3958333333333333, 0.4583332935969035,
1.2458333174387615, 0.0, 0.0, 0.0, 0.0, 0.004166666728754838,
0.0, 0.0, 0.0, 0.0, 0.0, 0.08333333333333333, 0.0, 0.0,
0.23749999205271402, 0.0833333432674408, 0.008333333457509676,
0.11666666467984517, 0.2916666865348816, 0.2208333412806193,
0.10000000397364299],
[0.0, 0.0, 0.0, 0.0, 0.0, 0.06666666766007741,
0.016666666915019352, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.041666666666666664, 0.0, 0.0, 0.0,
0.16666666666666666, 0.0, 0.0, 0.0, 0.0, 0.033333333830038704,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,
np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]])
def cor_single_vectors_x(data):
"""correct: reducing vectors to overlap."""
cor = np.ones((data.shape[0], data.shape[0]))
for inda in np.arange(data.shape[0]):
for indb in np.arange(data.shape[0]):
if indb > inda:
a, b = data[inda], data[indb]
m = ~a.mask & ~b.mask
sub = np.array([a.data[m], b.data[m]])
cor_coeff = np.corrcoef(sub)[0, 1]
cor[inda, indb] = cor_coeff
cor[indb, inda] = cor_coeff
return cor
def cor_single_vectors_ma_x_and_y(data):
"""correct: using masked vectors for x and y"""
cor = np.ma.ones((data.shape[0], data.shape[0]))
for inda in np.arange(data.shape[0]):
for indb in np.arange(data.shape[0]):
if indb > inda:
cor_coeff = np.ma.corrcoef(data[inda], data[indb])[0, 1]
cor[inda, indb] = cor_coeff
cor[indb, inda] = cor_coeff
return cor
def cor_single_vectors_ma_x(data):
cor = np.ma.ones((data.shape[0], data.shape[0]))
for inda in np.arange(data.shape[0]):
for indb in np.arange(data.shape[0]):
if indb > inda:
x = np.ma.array([data[inda], data[indb]])
cor_coeff = np.ma.corrcoef(x)[0, 1]
cor[inda, indb] = cor_coeff
cor[indb, inda] = cor_coeff
return cor
data = np.ma.masked_invalid(data)
cor = np.ma.corrcoef(data) # incorrect
cor_x = cor_single_vectors_x(data) # correct
cor_ma_x_and_y = cor_single_vectors_ma_x_and_y(data) # correct
cor_ma_x = cor_single_vectors_ma_x(data) # incorrect
# aside from the too large/small values
# cor[2,3] is incorrect
# cor_x[2,3] is correct
Error message:
No response
NumPy/Python version information:
1.21.2 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the np.ma.corrcoef call in the reproduction and compare it with cor_single_vectors_x and cor_single_vectors_ma_x_and_y using the supplied masked data. Trace the masked-array correlation implementation and add a regression case for the differing pairwise masks. Done means correlations match the overlapping-vector results and remain within [-1, 1].
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100