dask / dask/distributed

Support collection scatter

Open
#1,187 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm looking to broadcast the Futures in collectionB so that they can be involved in a reduction on each node. Pseudo-code illustrates the idea (one cannot serialise collectionB in client.submit for e.g.)

```python
def node_work(parallel_chunk, collectionB):
agg = 0
for serial_chunk in CollectionB:
agg += do_compute(parallel_chunk, serial_chunk)

return agg
agg = sum(as_completed(client.submit(node_work, parallel_chunk, collectionB)
for parallel_chunk in collectionA))
```

I think one can achieve a collection scattering effect using `client.replicate` but it blocks:

```python
CollectionA = client.persist(CollectionA)
CollectionB = client.persist(CollectionB)
client.replicate(CollectionB)

agg = do_compute(CollectionA, CollectionB)
```

It would be nice if one of the two were supported:

1. `scatter` supported collections as input, in which it would be possible to `dive` into the collection.
2. persist supported a `broadcast keyword`.

```python
CollectionA = client.persist(CollectionA)
CollectionB = client.scatter(CollectionB, broadcast=True)

agg = do_compute(CollectionA, CollectionB)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.