Matrix multiplication inconsistent for dask and sparse: `dask @ sparse` works, `sparse @ dask` fails
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
The order of matrix multiplication matters for dask and sparse arrays, but probably shouldn't.
Here is an example:
import numpy as np
from sparse import COO
import xarray as xr
sparse_da = xr.DataArray(COO.from_numpy(np.arange(10)))
dask_da = xr.DataArray(np.arange(100).reshape(10, 10).chunk({"dim_1": 5})
# this works:
dask_da @ sparse_da
# this raises a TypeError for "unsupported types"
sparse_da @ dask_da
# this works as expected
sparse_da @ dask_da.as_numpy()
In the workflow where this is used, the dask array has no chunks along its common dimensions with the sparse array, so it seems like sparse @ chunk should be fine. Also, in this workflow, loading the dask array into memory or making the sparse array dense would use a very large amount of memory.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reported example and comparing both matrix-multiplication orders. Trace how xarray DataArrays containing sparse and dask arrays handle each operand order; done means sparse_da @ dask_da works without loading the dask array into memory or densifying the sparse array.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100