dask.array.from_npy_stack doesn't support manually created chunks
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
I know this issue has been mentioned before ([here](https://stackoverflow.com/questions/50624710/dask-array-from-npy-stack-misses-info-file), @martindurant , but seems nothing came out of it in the end.
Happy to refactor into sth like this (if someone could comment if this looks good I'll issue a PR 😃 ) - I could also add some custom filename templates as well to make it more cross-compatible.
```python
def _from_npy_stack_meta(dirname, dtype, chunks, axis, mmap_mode="r"):
"""Load dask array from stack of npy files
:param dirname: the with the chunks. They must be labeled {ix}.npy
:param dtype: the dtype of the arrays
:chunks the size of the chunk across each chunk. n-dim tuple where only the axis dimension is non-singleton. E.g. ((10, 10, 5), (10,), (5,)) for 3-dim dask array with 3 chunks, split along axis 0.
:axis: the axis along the chunks have been saved.
"""
name = "from-npy-stack-%s" % dirname
keys = list(product([name], *[range(len(c)) for c in chunks]))
values = [
(np.load, os.path.join(dirname, "%d.npy" % i), mmap_mode)
for i in range(len(chunks[axis]))
]
dsk = dict(zip(keys, values))
return Array(dsk, name, chunks, dtype)
def from_npy_stack(dirname, mmap_mode="r"):
"""Load dask array from stack of npy files
Parameters
----------
dirname: string
Directory of .npy files
mmap_mode: (None or 'r')
Read data in memory map mode
See Also
--------
to_npy_stack
"""
with open(os.path.join(dirname, "info"), "rb") as f:
info = pickle.load(f)
return _from_npy_stack_meta(dirname, info["dtype"], info["chunks"], info["axis"], mmap_mode)
```
Contributor guide
Research direction
Start at dask.array.from_npy_stack and compare it with to_npy_stack, focusing on how the existing info data describes dtype, chunks, and axis. Done means manually created .npy chunks can be loaded without the info file while preserving the requested chunking and mmap behavior; the issue does not name tests or repository files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100