Dask coroutines
- Linguagem predominante
- Python
- Estrelas
- 1.7k
- Forks
- 778
- Merge médio
- 2h 50min
- PRs com merge (30d)
- 3
Descrição
There are currently a few ways to construct highly dynamic workloads, where the graph can change during computation. This includes operations like `get_client()`, calling `dask.compute` within a task, using futures and `as_completed`, and so on.
Sometimes these workloads can grow complex and difficult to reason about (see https://github.com/dask/distributed/issues/1424). Are there better programming interfaces to present to users that still cover the same options, but perhaps guide users to correct behavior.
Asynchronous projects like Tornado/Asyncio/Curio/Trio seem to prefer a coroutine-style approach. Is this a possible option for a distributed runtime like Dask? If so what would it look like?
Here are a couple of toy problems that come up frequently and naive thoughts on how they might look as coroutines
### Fibonacci
```python
@dask.coroutine
def fib(i):
if i < 2:
return i
else:
a, b = yield [fib(i - 1), fib(i - 2)]
return a + b
```
### Evaluating on a remote list of unknown size
```python
@dask.coroutine
def generate_data():
return list(range(random.randint(0, 10))) # a list of data of unknown length
@dask.coroutine
def inc(x):
return x + 1
@dask.coroutine
def my_len(L):
return len(L)
@dask.coroutine
def my_sum(L):
return sum(L)
@dask.coroutine
def process_all():
L = generate_data()
n = yield my_len(L)
processed = [inc(L[i]) for i in range(n)]
total = yield sum(processed)
return total
```
There are problems with both examples. They also don't represent the full space of complexity that existing solutions can cover. Broad thoughts on this topic are welcome.
cc @ogrisel @pitrou @remram44 @adamklein
Guia de contribuição
Direção de pesquisa
Start by reviewing the existing get_client(), dask.compute(), futures, and as_completed approaches described in the issue, along with distributed issue 1424. Compare them with the Fibonacci and remote-list examples to identify the required programming model. Done would require a concrete, agreed coroutine interface that covers these cases without the stated problems.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- distributed-systems
- Tipo de issue
- Funcionalidade
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Status de atividade
- Estagnada
- Clareza
- Precisa de esclarecimento
- Facilidade para iniciantes
- 20/100