dask / dask/distributed

Bundle gather calls in as_completed(..., with_results)

Open
#2,157 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently `as_completed(..., with_results=True)` can generate *many* gather calls, which can overwhelm the scheduler a bit. We might instead batch these gather calls if many come in before other gather calls return. This would also help to avoid creating very many file descriptors.

Alternativley this could also be handled at a lower level in gather itself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.