Set dask chunks when loading data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 10
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 5
Description
# ✨ Feature Request
It would be beneficial to choose dask chunks on load to take advantage of the chunking already on disk. These resources may help:
1. [https://docs.dask.org/en/stable/array-chunks.html#chunks](https://docs.dask.org/en/stable/array-chunks.html#chunks)
2. [https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes](https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes)
In particular, from the rules of thumb in 2., we have:
> Some people have observed that chunk sizes below 1MB are almost always bad. Chunk size between 100MB and 1GB are generally good, going over 1 or 2GB means you have a really big dataset and/or a lot of memory available per core,
In the case of SPICE, we do have a lot of memory available per core, so may want to take advantage of larger chunks.
In the case of netCDF source files, we probably want the ultimate chunk sizes to be integer multiples of the netCDF chunk sizes on disk.
## Context
Note iris load does now have capability for setting dask chunks on load, so this may be exposing the existing iris capability. I would not be surprised if there isn't additional complexity, though. It may be wise to re-scope this down to `Set dask chunks when loading data from netCDFs`, and having follow up issues for other formats.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining how ANTS loads data through Iris and how Iris exposes dask chunk selection, then review the linked Dask chunking guidance. Define the initial scope around netCDF sources, with completion requiring configurable load-time chunks that can align with on-disk netCDF chunk sizes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100