pydata / pydata/sparse

Poor numerical consistency with numpy for reductions with dtype=`float16` specified

Open
#191 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
668
Forks
141
Avg merge
2d 8h
Merged PRs (30d)
4

Description

Reductions like COO.sum and COO.mean fail to match numpy in all cases when dtype=numpy.float16 is specified. For example:

import sparse
import numpy as np
x = np.array([[[    0,     0,  4526,     0],
               [    0,     0,   -37,     0],
               [ 8372,     0,  7915,     0]],
              [[    0,     0,     0,     0],
               [    0,     0, -7917,     0],
               [-9719,     0,     0,     0]]], dtype='i4')

s = sparse.COO.from_numpy(x)
res = s.sum(axis=(0, 2), dtype='f2')
sol = x.sum(axis=(0, 2), dtype='f2')
print(res.todense())
print(sol)

outputs:

[ 4530. -7950.  6564.]
[ 4530. -7950.  6570.]

It's not clear if each of these will need to be fixed per-method, or if there's a general fix in the reduce code. For sum, numpy interprets x.sum(dtype='f2') the same as x.astype('f2').sum(), but this is not true for x.mean(dtype='f2').

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the COO.sum and COO.mean reduction entry points and reproduce the provided float16 examples against NumPy. Trace how the dtype argument is handled for reductions over multiple axes, then verify that sparse results match the corresponding NumPy results for the shown cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.