Support collection scatter
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I'm looking to broadcast the Futures in collectionB so that they can be involved in a reduction on each node. Pseudo-code illustrates the idea (one cannot serialise collectionB in client.submit for e.g.)
```python
def node_work(parallel_chunk, collectionB):
agg = 0
for serial_chunk in CollectionB:
agg += do_compute(parallel_chunk, serial_chunk)
return agg
agg = sum(as_completed(client.submit(node_work, parallel_chunk, collectionB)
for parallel_chunk in collectionA))
```
I think one can achieve a collection scattering effect using `client.replicate` but it blocks:
```python
CollectionA = client.persist(CollectionA)
CollectionB = client.persist(CollectionB)
client.replicate(CollectionB)
agg = do_compute(CollectionA, CollectionB)
```
It would be nice if one of the two were supported:
1. `scatter` supported collections as input, in which it would be possible to `dive` into the collection.
2. persist supported a `broadcast keyword`.
```python
CollectionA = client.persist(CollectionA)
CollectionB = client.scatter(CollectionB, broadcast=True)
agg = do_compute(CollectionA, CollectionB)
```
Contributor guide
Assessment
This issue has not been assessed yet.