reshape fails when the chunk size is larger than limit size
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
In my code, I try to reshape the array ```Y``` from shape ```(36,36,36,468)``` to ```(46656,468)``` and limit the max size to ```32MiB```. In this case, the chunk size of ```Y``` is ```(18,18,36,468)->(9,36,36,468)->(11664,468)```, which is larger than ```32MiB```(because the data type is complex128), so the chunk of ```Y``` is splited to ```(4266,468)```$\times 10$ and ```(3996,468)```$\times 1$. But when we compute or persist ```Y```, the following error arise:
```ValueError: cannot reshape array of size 5458752 into shape (4266,468)```
It seems that the dask array is trying to reshape the chunk ```(11664,468)``` to ```(4266,468)``` but not split.
**Minimal Complete Verifiable Example**:
```python
import dask.array as da
import numpy as np
Dcut=36
l=Dcut*13
N=int(Dcut**3)
n=2
rs = da.random.RandomState(seed=1234, RandomState=np.random.RandomState)
chunks_Y=(Dcut/n, Dcut/n, Dcut, l)
Y = rs.normal(size=(Dcut,Dcut,Dcut,l), chunks=chunks_Y).astype(np.complex128)
Y = da.reshape(Y, shape=(N,l), limit='32M')
Y=Y.persist()
```
**Anything else we need to know?**:
**Environment**:
- Dask version: 2023.1.1
- Python version: 3.10
- Operating System: Red Hat
- Install method (conda, pip, source): conda-forge
Contributor guide
Assessment
This issue has not been assessed yet.