dask / dask/dask

reshape fails when the chunk size is larger than limit size

Open
#10,199 0 comments 0 reactions 0 assignees View on GitHub
array needs attention needs triage
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
In my code, I try to reshape the array ```Y``` from shape ```(36,36,36,468)``` to ```(46656,468)``` and limit the max size to ```32MiB```. In this case, the chunk size of ```Y``` is ```(18,18,36,468)->(9,36,36,468)->(11664,468)```, which is larger than ```32MiB```(because the data type is complex128), so the chunk of ```Y``` is splited to ```(4266,468)```$\times 10$ and ```(3996,468)```$\times 1$. But when we compute or persist ```Y```, the following error arise:
```ValueError: cannot reshape array of size 5458752 into shape (4266,468)```
It seems that the dask array is trying to reshape the chunk ```(11664,468)``` to ```(4266,468)``` but not split.

**Minimal Complete Verifiable Example**:

```python
import dask.array as da
import numpy as np
Dcut=36
l=Dcut*13
N=int(Dcut**3)
n=2
rs = da.random.RandomState(seed=1234, RandomState=np.random.RandomState)

chunks_Y=(Dcut/n, Dcut/n, Dcut, l)
Y = rs.normal(size=(Dcut,Dcut,Dcut,l), chunks=chunks_Y).astype(np.complex128)
Y = da.reshape(Y, shape=(N,l), limit='32M')
Y=Y.persist()
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2023.1.1
- Python version: 3.10
- Operating System: Red Hat
- Install method (conda, pip, source): conda-forge

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.