dask chunking tutorial outline
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- data, documentation
Research direction
Review the linked Dask chunking resources and the repository's existing tutorial structure first; no target file is named in the issue. Done means an intermediate Xarray-user tutorial covers chunk-size effects, monitoring chunks and tasks, rechunking cost, and read-time versus storage chunk choices, including the proposed examples and images.
Written by the indexing model from the issue text.
Description
from the pangeo working meeting discussion with @mgrover1 @jmunroe @norlandrhagen
Here's an outline for an intermediate tutorial talking about dask chunking specifically for Xarray users
Motivation: why care about chunk size?
- demonstrate relation between chunk size and computation time / number of tasks with a simple example?
- maybe even memory usage
- https://tutorial.dask.org/02_array.html#Choosing-good-chunk-sizes
- https://docs.dask.org/en/stable/array-chunks.html
Keeping track
- monitoring chunk sizes and num tasks throughout the pipeline using the repr
- use some images
- while output blocks may be small (say after a big reduction), intermediate blocks need not be.
- So keep monitoring chunksizes (and tasks) throughout the pipeline.
Why is it important to choose appropriate chunks early in the pipeline?
- Demonstrate that rechunking is not cheap in most cases
Specify chunks when reading data
- Avoid
chunks="auto". - Specifying
chunksduring data readopen_datasetopen_mfdataset
- Analysis vs storage chunks:
- Dask chunks should be a multiple of chunks on disk
- talk about aligning chunks with files stored on disk
- @djhoese example
- Dominant language
- Jupyter Notebook
- Stars
- 204
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from xarray-contrib/xarray-tutorial
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
help wanted
Difficulty 4/5 3-5 days Newbie friendliness 48/100
xarray-contrib/xarray-tutorial#362 · 2 comments · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
All issues in xarray-contrib/xarray-tutorial
Similar issues
-
check:passed logos:remove
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
bug needs triage p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
GoogleCloudPlatform/DataflowTemplates#4273 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
correction metadata
Difficulty 1/5 Under an hour Newbie friendliness 72/100
acl-org/acl-anthology#10025 · 1 comment ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100