dask / dask/distributed

Task Group visualization doesn't scale to thousands of task groups

Open
#5,021 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I just saw an example where there were a thousand TaskGroups, each of size one. This is arguably the fault of the code that produced such a wonky setup, but we should be robust to it.

As a solution, I propose that if there are more than, say, 200 TaskGroups, that we only display those that have many tasks

Actually, that doesn't quite work for a few reasons:

1. We may lose important dependencies like `sum()`
2. That may break parts of the graph, resulting in odd holes, killing our layout algorithm

But! Maybe we can be clever here in some way! Thoughts welcome. cc @ncclementi @jrbourbeau

In the specific case that I saw the thousand single-task groups were all data creation tasks without dependencies. @ian-r-rose may be able to provide a reproducer.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.