dask / dask/dask

Error using db.from_delayed with sparse arrays

Open
#11,643 2 comments 0 reactions 0 assignees View on GitHub
bug delayed
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

```
File "python3.11/site-packages/dask/bag/core.py", line 1881, in reify
if len(seq) and isinstance(seq[0], Iterator):
^^^^^^^^^^^^^^^^^
File "python3.11/site-packages/scipy/sparse/_base.py", line 425, in __len__
raise TypeError("sparse array length is ambiguous; use getnnz()"
```

**Minimal Complete Verifiable Example**:

```python
import dask.bag as db
import numpy as np
from dask import delayed
from scipy.sparse import csr_array

def add(x, y):
return x + y

@delayed
def create_sparse_array_delayed():
return csr_array(np.random.random((10, 10)))

@delayed
def create_array_delayed():
return np.random.random((10, 10))

db.from_sequence(
[csr_array(np.random.random((10, 10))), csr_array(np.random.random((10, 10)))]).fold(
add).compute() # works with sparse arrays when created from sequence
db.from_delayed([create_array_delayed(), create_array_delayed()]).fold(add).compute() # works with numpy arrays
db.from_delayed([create_sparse_array_delayed(), create_sparse_array_delayed()]).fold(add).compute() # fails

```

**Environment**:

- Dask version: 2024.12.0
- Python version: 3.11
- Operating System: Mac
- Install: pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.