Blockwise optimization doesn't combine task names, like low-level fusion does
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
Users may be confused why tasks that they expect to run don't show up on the dashboard.
With low-level fusion, we concatenate the names of all the tasks together, so at least the name of the task (like `read_csv-assign-getitem-add`) gives you some hint that all those things are still happening. But with blockwise fusion, that same task would just be named `add`, which gives you no indication that your `read_csv`, `assign`, or `getitem` are contained within it.
```python
In [1]: import dask.dataframe as dd
# long, all-blockwise operation
In [2]: df = dd.demo.make_timeseries().assign(z=lambda df: df.x + df.y).pipe(lambda df: df[["x", "y", "z"]] + 1)
In [3]: import distributed
In [4]: client = distributed.Client()
In [4]: pdf = df.persist()
```

On the dashboard, the tasks just show up as `add`, so you might be confused what happened to all the other operations.
This could arguably be solved with better documentation (xref https://github.com/dask/dask/issues/8627), but I do think having the concatenated names is more user-friendly than having to find some obscure docs to understand this.
However, I recognize that if you change the name of the layer, you also have to change the names referenced in the dependent layers. That's just not currently possible with HLG layers—we could feasibly do it to MaterializedLayers, but in general, there's no `rename_input_key` interface on a Layer.
Furthermore, this is actually slightly violating the scheduler's assumptions about TaskPrefix keys. In my example above, the scheduler is going to learn a duration estimate for `add` tasks which actually includes that whole chain of operations, and will be quite wrong if I later submit `(df + 1).compute()`.
Maybe we could add a `task-group-key` and `task-prefix-key` annotation to the rewritten layer, which the scheduler would recognize and use instead of `key_split` and `key_split_group` (and would ultimately be displayed on the dashboard)?
cc @rjzamora @jrbourbeau
Contributor guide
Assessment
This issue has not been assessed yet.