pydata / pydata/sparse

Some dtypes not supported by numba

Open
#521 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

upstream
Dominant language
Python
Stars
668
Forks
141
Avg merge
2d 8h
Merged PRs (30d)
4

Description

Describe the bug
Big-endian floats (and maybe others) are not supported by numba, which leads to TypingError on some sparse operations.

To Reproduce

import numpy as np
import sparse as sp

little = np.array([1, 1, 1, 1, 1, 1], dtype='<f8')
big = np.array([1, 1, 1, 1, 1, 1], dtype='>f8')

Anp = np.random.randint(100, size=(4, 6)) / 100
A = sp.COO.from_numpy(Anp)

A.dot(little) # works
A.dot(big) # fails

The other way around is the same;

Abig = sp.COO.from_numpy(Anp.astype('>f8'))

Abig.dot(little) # fails
Abig.dot(big) # fails

Expected behavior
I expected the operation to work in all cases.

Could on-the-fly byte swap be implemented? May be with a warning?

In this specific case, on-the-fly byte swapping is what numpy does:

r = Anp.dot(big)  
r.dtype # <f8, little-endian

For the case with Abig, where the sparse array itself is in a unsupported dtype, if not on-the-fly conversion is done, maybe there could be an issue raised, or at least a warning, on creation? To signify that the input array doesn't support the full features of sparse.

System

  • OS and version: Ubuntu 18.04
  • sparse version : 0.13.0
  • NumPy version : 1.21.2
  • Numba version : 0.53.1

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python reproduction with NumPy 1.21.2, Numba 0.53.1, and sparse 0.13.0 to compare little- and big-endian inputs in COO dot operations. Trace the resulting TypingError and determine the supported behavior for byte-swapping or rejecting unsupported dtypes; done means the reported cases behave consistently with the agreed warning or error behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.