carpentries-incubator / carpentries-incubator/lesson-parallel-python

update dask delayed chapter

Open
#144 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Dockerfile
Stars
11
Forks
15
Avg merge
3m
Merged PRs (30d)
3

Description

We could motivate the introduction of `dask.delayed` better by also introducing `asyncio`, `concurrent.futures` and `dask.futures` shortly. The advantage of `dask.delayed` is that we can visualize the call graph easily, and make visible what is happening. What all of these ideas have in common, is that they operate in a universe of "coloured" functions. If you want to call a delayed function, you receive a delayed object or future, if you want to unwrap this object, you have two choices, make the new function also delayed or run the computation (unwrapping). In this sense there is a sort of viral nature to all of these.

Still there are some major differences: `dask.delayed` is lazy, while most future implementations are eager; `concurrent.futures` runs using an `Executor` which can be multi-threading or multi-processing, while `dask.futures` also run distributed. Futures can be cancelled.

This issue replaces #82

Contributor guide

Open the contributing guide

Research direction

Start by locating the delayed chapter in the lesson and read the existing introduction to dask.delayed. Add a brief comparison with asyncio, concurrent.futures, and dask.futures, covering laziness, execution models, distribution, cancellation, and call-graph visualization; the chapter is done when these differences and shared future-like behavior are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.