dask chunking tutorial outline
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 文档
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 领域
- data, documentation
调研方向
Review the linked Dask chunking resources and the repository's existing tutorial structure first; no target file is named in the issue. Done means an intermediate Xarray-user tutorial covers chunk-size effects, monitoring chunks and tasks, rechunking cost, and read-time versus storage chunk choices, including the proposed examples and images.
由索引模型根据 Issue 内容生成。
描述
from the pangeo working meeting discussion with @mgrover1 @jmunroe @norlandrhagen
Here's an outline for an intermediate tutorial talking about dask chunking specifically for Xarray users
Motivation: why care about chunk size?
- demonstrate relation between chunk size and computation time / number of tasks with a simple example?
- maybe even memory usage
- https://tutorial.dask.org/02_array.html#Choosing-good-chunk-sizes
- https://docs.dask.org/en/stable/array-chunks.html
Keeping track
- monitoring chunk sizes and num tasks throughout the pipeline using the repr
- use some images
- while output blocks may be small (say after a big reduction), intermediate blocks need not be.
- So keep monitoring chunksizes (and tasks) throughout the pipeline.
Why is it important to choose appropriate chunks early in the pipeline?
- Demonstrate that rechunking is not cheap in most cases
Specify chunks when reading data
- Avoid
chunks="auto". - Specifying
chunksduring data readopen_datasetopen_mfdataset
- Analysis vs storage chunks:
- Dask chunks should be a multiple of chunks on disk
- talk about aligning chunks with files stored on disk
- @djhoese example
- 主要语言
- Jupyter Notebook
- 星标
- 204
- 派生
- 121
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
xarray-contrib/xarray-tutorial 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 75/100
-
help wanted
难度 4/5 3-5 天 新手友好度 48/100
xarray-contrib/xarray-tutorial#362 · 2 条评论 · 1 个 reaction ·
-
难度 2/5 1-3 小时 新手友好度 55/100
-
难度 5/5 一周以上 新手友好度 35/100
-
难度 4/5 3-5 天 新手友好度 38/100
查看 xarray-contrib/xarray-tutorial 的全部 Issue
相似的 Issue
-
bug needs triage p2
难度 2/5 1-3 小时 新手友好度 78/100
GoogleCloudPlatform/DataflowTemplates#4273 · 1 条评论 ·
-
难度 1/5 1 小时以内 新手友好度 90/100
-
correction metadata
难度 1/5 1 小时以内 新手友好度 72/100
acl-org/acl-anthology#10025 · 1 条评论 ·
-
bug
难度 2/5 1-3 小时 新手友好度 88/100
-
难度 2/5 1-3 小时 新手友好度 76/100